Introduction
Any production system should include a provision for taking regular backups. Vault Enterprise can be configured to take and store snapshots at a specific interval.
Prerequisites
Vault Enterprise deployed with Integrated Storage (Raft)
Problem
Configure Raft automated snapshots against Google GCS using JSON data and erroneously include "google_endpoint":false. Example command:
curl -k --location 'http://127.0.0.1:8200/v1/sys/storage/raft/snapshot-auto/config/test' \
--header 'X-Vault-Token: hvs.Uvaxxxxxxxx' \
--header 'Content-Type: text/plain' \
--data '{
"file_prefix": "vault-snapshot",
"google_disable_tls": false,
"google_endpoint": false,
"google_gcs_bucket": "vaulttest",
"google_service_account_key": "",
"interval": 15,
"path_prefix": "snapshots/",
"retain": 7,
"storage_type": "google-gcs"
}'
This results in a failure to store the snapshot in Google GCS. Example of error observed in Vault operational logs, notice the https://0/ URL:
[ERROR] core.snapshotmgr.test: snapshot failure: name=test error="error listing current snapshots: Get \"https://0/storage/v1/b/vaulttest/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=&prettyPrint=false&projection=full&startOffset=&versions=false\": dial tcp: lookup 0: no such host" consecutive errors=1 next retry=15s
Cause
While generally the google_endpoint parameter is not set, if exporting the config detail (vault read ...) and then importing that again into Vault causes this to incorrectly set the google_endpoint to 0.
Solution
Not setting the parameter "google_endpoint" resolves the issue.
Additional References
Vault Documentation: Automated integrated storage snapshots
Vault Documentation: Automated snapshots API