In Vault Enterprise, it's possible to encounter issues when a snapshot is downloaded from a Vault node UI. This occurs when the snapshot request is routed to a standby node rather than the active leader. Performance standby nodes are not designed to generate snapshots and may return incomplete or invalid files.
In this case, the downloaded
snapshot.gz is very small (just a few bytes in size) and unusable. For example, the snapshot size is only 9 bytes like below:To verify the snapshot, run the snapshot inspection command against snapshot.gz file:
vault operator raft snapshot inspect snapshot.gzThe output shows an error like below:
Error reading snapshot: failed to decompress snapshot: unexpected EOFThis indicates that the snapshot is corrupted or incomplete. This is a known limitation in the product is common to all versions including Vault 1.21.0 and prior.
Resolution and Workaround
- Always download snapshots directly from the current leader node, either through its UI or API.
- Ensure your routing or load balancer configuration directs snapshot-related requests exclusively to the leader node.
- Consider creating a separate DNS entry dedicated to administrative tasks (such as snapshot operations) that routes only to the leader node.