Introduction
There are multiple ways to automate integrated storage snapshots. One storage type with the parameter explained in this article is local storage type. A required parameter for the local storage type is local_max_space. This will set the maximum space, in bytes, to use for all snapshots with the given file_prefix in the path_prefix directory. This article will explain the how to troubleshoot an automated snapshot error stating "exceeded max space" error.
Use Case
Below is an example of a local, automated snapshot configuration:
Key Value
--- -----
file_prefix config1
interval 43200
local_max_space 1000000
path_prefix /opt/vault/snapshots
retain 10
storage_type local
Note the file_prefix is set to config1 and the path_prefix is the /opt/vault/snapshots directory, with the local_max_space set to 1000000 bytes.
If we look at a successful automated snapshot, we can see the size is 140288, which is well below the set local_max_space.
[DEBUG] core.snapshotmgr.config1: snapshot complete: name=config1 elapsed=92.825333ms size=140288
As the Vault database grows and/or automated snapshots are saved over time, it is possible to encounter the following error:
[ERROR] core.snapshotmgr.config1: snapshot failure: name=config1
error=
| 3 errors occurred:
| \t* failed to write snapshot file: failed to write snapshot metadata: io: read/write on closed pipe
| \t* exceeded max space: written=17408, available=17353, max_space=1000000
| \t* exceeded max space: written=17408, available=17353, max_space=1000000
|
consecutive errors=1 next retry=10s
The local_max_space parameter takes into account all snapshots with the same file_prefix and path_prefix defined. The local_max_space value will need to be greater than the total of all snapshots stored from a single automated configuration.
If the Day 1 snapshot size is 140288 and we plan on retaining 10 snapshots, we would need the local_max_space parameter to be set to at least 1402880 bytes. However, with growth to Vault and a growth in the database size, this value will need to be increased over time. To properly set the local_max_space parameter, we will need to take into account items that lead to growth in Vault and set this parameter accordingly.
Additional Information
Automated Snapshots Docs: https://developer.hashicorp.com/vault/docs/enterprise/automated-integrated-storage-snapshots
Automated Snapshots API Docs: https://developer.hashicorp.com/vault/api-docs/system/storage/raftautosnapshots