Attempting to manually create a Vault snapshot when using the Raft/Integrated Storage backend may fail with the message
Error taking snapshot: incomplete snapshot, unable to read SHA256SUMS.sealed file.
This issue occurs when the request to create a snapshot is handled by a standby or Performance Standby node in the Vault cluster. At present when a snapshot request is received by a node which is not the active / leader node, it will fail with this error as only the active node is capable of generating snapshots.
In order to take a manual snapshot using the
vault operator raft snapshot save command the request must be directed to the active node in the cluster.
The active node can be identified by looking for the node reporting the value of
leader in the output of
vault operator raft list-peers, or the node reporting the value of
Active Node as
true in the output of
vault operator members.
Once the active node is identified some of the available options for taking a snapshot include:
- Prefixing the command with the
VAULT_ADDRenvironment variable, for example:
VAULT_ADDR=https://node754.stg.vault.nicecorp.org:8200 vault operator raft snapshot save my-snapshot.snap. This instructs the Vault process to direct the request to the active node.
- Opening a terminal session on the active node and running the
vault operator raft snapshot savecommand.
- Configure a separate service / VIP on a load balancer which directs all requests to the current active node in the cluster, and route the snapshot save command to this service / VIP using the
VAULT_ADDRenvironment variable. This can be a suitable option when direct access to the nodes is not available, and requests musts be routed via a load balancer.
When the active node is directly targeted in the snapshot save request, the manual snapshot creation process succeeds.
An internal issue with reference VAULT-4568 has been created for this issue and is being considered for inclusion in a future Vault release. Enterprise customers can quote this reference to get status updates on the issue.
vault operator raft snapshot savecommand: https://developer.hashicorp.com/vault/docs/commands/operator/raft#snapshot-save
- GitHub Issue: https://github.com/hashicorp/vault/issues/15258