If your Vault data are stored in Consul’s key/value (KV) storage, you might also be using the Consul Snapshots to take backups of the Consul state which will include the Vault data. As part of your backup process you might also need to verify the snapshot integrity.
For Consul Snapshots, there are some limited direct means of snapshot verification.
Verify Integrity of Consul Snapshot File
The snapshot data are read back to verify before writing out to the snapshot file within the snapshot_save.go code, and this same type of verification is also available after the file is written through the consul snapshot inspect CLI, which produces some details about the Consul snapshot like this:
$ consul snapshot inspect example.snap
ID 4-18122-1562594496718
Size 897824
Index 18122
Term 4
Version 1
Interpretation of the output is documented in the Consul Snapshot Inspect documentation:
- ID - A unique ID for the snapshot, only used for differentiation purposes
- Size - The size of the snapshot, in bytes.
- Index - The Raft index of the latest log entry in the snapshot.
- Term - The Raft term of the latest log entry in the snapshot.
- Version - The snapshot format version. This only refers to the structure of the snapshot, not the data contained within.
The snapshot file format itself is essentially just a Tarball, so you can also unarchive and further inspect it:
$ tar zxvf example.snap
x meta.json
x state.bin
x SHA256SUMS
A Consul snapshot contains the 3 files shown in the example output above.
The meta.json
file represents Consul cluster configuration at the time of the snapshot; here’s an example:
{
"Version": 1,
"ID": "4-18122-1562594496718",
"Index": 18122,
"Term": 4,
"Peers": "k68xNzIuMTcuMC4yOjgzMDCvMTcyLjE3LjAuNDo4MzAwrzE3Mi4xNy4wLjM6ODMwMA==",
"Configuration": {
"Servers": [
{
"Suffrage": 0,
"ID": "c0ffee74-a33e-4200-99ae-12dc45a4a6ae",
"Address": "172.17.0.2:8300"
},
{
"Suffrage": 0,
"ID": "c0ffee74-77f0-44ea-849a-4bfeef9b07c4",
"Address": "172.17.0.4:8300"
},
{
"Suffrage": 0,
"ID": "c0ffee74-cb59-4bec-9eba-ca4a3fe56646",
"Address": "172.17.0.3:8300"
}
]
},
"ConfigurationIndex": 1,
"Size": 897824
}
The state.bin
is a binary representation of the RAFT data from the Consul cluster, which in turn contains all data, including the Vault data.
Finally, there is a SHA256SUMS
file which contains the SHA 256 summaries of the snapshot files:
$ cat SHA256SUMS
4549f928ae2420ee3f7243a01c71a90e16c2e0fc1ec7896b5cbb050a6b26f40a meta.json
ad6658de3086c16297b668028a23f1796111904ffb3b4e424858778b7785d6a5 state.bin
Using this information, you can perform some rudimentary validation of the snapshot data integrity from the perspective of the Consul server cluster and ensure the files you are dealing with are correct and as originally written by Consul during the snapshot operation.
Restore Snapshot and Directly Inspect Consul KV
You can also restore the Consul snapshot and inspect the Vault data in Consul’s key/value store to a limited extent. See Inspecting Vault Data for more details on how you can list the key/value paths and keys in Consul for relevant Vault secrets and leases.
This will allow for verification that secrets, leases, and tokens are present and other details such as their total count without the need to spin up a Vault server for more intensive validation.
Since Vault key/value data in Consul is encrypted and cannot be fully verified with respect to correct values outside of Vault, you would need to restore the snapshot into a Consul environment, then point a Vault server to that environment, unseal Vault, and use whatever means you prefer (scripting, etc.) to verify the contents of Vault data to the satisfaction of your organizational standards.