Vault snapshot save fails with message: "Error taking snapshot: incomplete snapshot, unable to read SHA256SUMS.sealed file" – HashiCorp Help Center

Introduction

Problem

Attempting to manually create a Vault snapshot when using the Raft/Integrated Storage backend may fail with the message Error taking snapshot: incomplete snapshot, unable to read SHA256SUMS.sealed file.

Cause

This issue occurs when the request to create a snapshot is handled by a standby or Performance Standby node in the Vault cluster. At present when a snapshot request is received by a node which is not the active / leader node, it will fail with this error as only the active node is capable of generating snapshots.

Solution

In order to take a manual snapshot using the vault operator raft snapshot save command the request must be directed to the active node in the cluster.

The active node can be identified by looking for the node reporting the value of State as leader in the output of vault operator raft list-peers, or the node reporting the value of Active Node as true in the output of vault operator members.

Once the active node is identified some of the available options for taking a snapshot include:

Prefixing the command with the VAULT_ADDR environment variable, for example: VAULT_ADDR=https://node754.stg.vault.nicecorp.org:8200 vault operator raft snapshot save my-snapshot.snap. This instructs the Vault process to direct the request to the active node.
Opening a terminal session on the active node and running the vault operator raft snapshot save command.
Configure a separate service / VIP on a load balancer which directs all requests to the current active node in the cluster, and route the snapshot save command to this service / VIP using the VAULT_ADDR environment variable. This can be a suitable option when direct access to the nodes is not available, and requests musts be routed via a load balancer.

Outcome

When the active node is directly targeted in the snapshot save request, the manual snapshot creation process succeeds.

An internal issue with reference VAULT-4568 has been created for this issue and is being considered for inclusion in a future Vault release. Enterprise customers can quote this reference to get status updates on the issue.

Additional Information

vault operator raft snapshot save command: https://developer.hashicorp.com/vault/docs/commands/operator/raft#snapshot-save
GitHub Issue: https://github.com/hashicorp/vault/issues/15258