Verify Consul Snapshot Integrity

If Vault is using Consul as the storage backend, Consul snapshots can be used to take backups of the Consul state, which will include Vault data. You can verify the integrity of snapshots using the following instructions in this article.

Verify Integrity of Consul Snapshot File

The snapshot data is first read back for verification prior to being written to the snapshot file within the snapshot_save.go code. This verification is also available after the file is written using the consul snapshot inspect command, which provides information about the Consul snapshot.

$ consul snapshot inspect <file_name>.snap
ID           4-18122-1562594496718
Size         897824
Index        18122
Term         4
Version      1

Interpretation of the output is documented in the Consul Snapshot Inspect documentation:

• ID - A unique ID for the snapshot, only used for differentiation purposes
• Size - The size of the snapshot, in bytes
• Index - The Raft index of the latest log entry in the snapshot
• Term - The Raft term of the latest log entry in the snapshot
• Version - The snapshot format version. This only refers to the structure of the snapshot, not the data contained within.

The snapshot file is stored as a tarball, which can be unarchived for inspection.

$ tar zxvf <file_name>.snap
meta.json
state.bin
SHA256SUMS

A Consul snapshot contains the 3 files shown in the example output above.

The meta.json file represents the Consul cluster configuration at the time of the snapshot. Here’s an example:

{
  "Version": 1,
  "ID": "4-18122-1562594496718",
  "Index": 18122,
  "Term": 4,
  "Peers": "k68xNzIuMTcuMC4yOjgzMDCvMTcyLjE3LjAuNDo4MzAwrzE3Mi4xNy4wLjM6ODMwMA==",
  "Configuration": {
    "Servers": [
      {
        "Suffrage": 0,
        "ID": "c0ffee74-a33e-4200-99ae-12dc45a4a6ae",
        "Address": "172.17.0.2:8300"
      },
      {
        "Suffrage": 0,
        "ID": "c0ffee74-77f0-44ea-849a-4bfeef9b07c4",
        "Address": "172.17.0.4:8300"
      },
      {
        "Suffrage": 0,
        "ID": "c0ffee74-cb59-4bec-9eba-ca4a3fe56646",
        "Address": "172.17.0.3:8300"
      }
    ]
  },
  "ConfigurationIndex": 1,
  "Size": 897824
}

The state.bin is a binary representation of the RAFT data from the Consul cluster, which in turn contains all data, including Vault data.

Finally, there is a SHA256SUMS file which contains the SHA 256 summaries of the snapshot files:

$ cat SHA256SUMS
4549f928ae2420ee3f7243a01c71a90e16c2e0fc1ec7896b5cbb050a6b26f40a  meta.json
ad6658de3086c16297b668028a23f1796111904ffb3b4e424858778b7785d6a5  state.bin

Using this information, you can perform some rudimentary validation of the snapshot data integrity from the perspective of the Consul server cluster and ensure the files you are dealing with are correct and as originally written by Consul during the snapshot operation.

Restore Snapshot and Directly Inspect Consul KV

You can also restore the Consul snapshot and inspect the Vault data in Consul’s key/value store to a limited extent. See Inspecting Vault Data for more details on how you can list the key/value paths and keys in Consul for relevant Vault secrets and leases.

This enables you to verify secrets, leases, tokens, and other additional details such as total count, without having to spin up a Vault server for more comprehensive validation.

Since Vault key/value data in Consul is encrypted and cannot be fully verified with respect to correct values outside of Vault, you would need to restore the snapshot into a Consul environment, point a Vault server to that environment, unseal Vault, and utilize a method such as scripting or other tools, to verify the contents of the Vault data.

Resources

1. Consul Snapshot
2. Tarball
3. snapshot_save.go
4. Inspecting Vault Data