Consul snapshots & Vault mount restoration via Consul KV export / import

Avatar
Mehdi Ahmadi
  • Updated

This article makes direct use of Vaults Consul storage & internal properties that should be pursued only after provisions for backup as well as rollback have been made in case of any issues. Additional requirements include Vault recovery keys, configuration parameter raw_storage_endpoint and a token with a privileged policy on the path of: sys/*

There's no feature in Vault to allow for selective restoration of particular mounts, namespaces or other specifics from any of the various storage types that are supported in Vault (Consul, Integrated Storage, etc). However, it may be possible to leverage Consul snapshots with the steps outlined in this article to recover particular Vault secrets or authentication mounts that were inadvertently disabled and which now need to be restored.

After restoring the Consul snapshot containing the deleted mount onto a separate Consul instance, the Consul KV export (consul kv export) / import features as well as Vault recovery-mode can be used to export mounts that can then be imported on the impacted Consul & Vault cluster where they are required.

A final restart to Vault is performed when Consul import is done and Vaults internal configuration edited to re-include the mount that was imported; the final result is a Vault instance with mount that was previously unavailable

Restoring Consul Snapshot, Consul Exporting & Vault reading past

The Consul snapshot containing the disabled mounts should be restored onto a temporary instance which will be used to export Consul KV values related to the mount in question as well as starting Vault in recovery-mode (-recovery-mode).

The steps below commences on the premise that a Consul restoration has been made and that the Vault executables are also present on the same host with the user already in the Vault recovery-mode.

The UID of the mount that's sought after and its detailed that were removed should be read from the restored snapshot. The CLI example below uses jq to read details of all mounts at the time when the snapshot was made:

vault read -format=json sys/raw/core/mounts | jq .'data.value' \
  | sed 's/\\"/"/g' | sed 's/\"{/\{/g' | sed 's/}\"/}/g' \
  | jq > mounts_deleted.json

The UID of the mount which was deleted can be found in the output above and in this example value of c2cbd2c9-33cd-dcf5-3cb3-3680cce6416a presents a KV2 Secrets engine that will be exported from vault/logic/c2cbd2c9-33cd-dcf5-3cb3-3680cce6416a:

# // Vault logical path for mount data
VUID=c2cbd2c9-33cd-dcf5-3cb3-3680cce6416a

# // to list / view all available mounts at the time of snapshot:
consul kv get -keys vault/logical/

# // export mount for later import
consul kv export vault/logical/${VUID} > consul_kv_restore.json

Consul Importing

The previously gathered files (mounts_deleted.json & consul_kv_restore.json) should be available on the target Vault environment where they're required.

The Consul KV data of the mount that was exported from the prior restoration instance can be imported using:

# // list current mounts prior to import:
consul kv get -keys vault/logical/

# // import:
consul kv import @consul_kv_restore.json

# // re-confirm import was successful
consul kv get -keys vault/logical/

This imports the correlating path (vault/logic/c2cbd2c9-33cd-dcf5-3cb3-3680cce6416a) and all it's sub-values that was previously exported.

Updating Vault Mounts definition & restarting Vault

The next step is to update the internal configuration document in Vault that's responsible for all mount definitions. If the (HCL) configuration parameter raw_storage_endpoint is already enabled then these operations can be performed while the target Vault instances is up and running in a regular service state without the need for recovery-mode.

Reading the current mount definitions, from: sys/raw/core/mounts and comparing it with what was previously exported can help determine the deleted mount as well as differences of the code block which need to be re-authored back.

# // read current mount definitions
vault read -format=json sys/raw/core/mounts | jq .'data.value' \
  | sed 's/\\"/"/g' | sed 's/\"{/\{/g' | sed 's/}\"/}/g' \
  | jq > mounts_current.json

# // show differences between old & new
diff -y <(jq --sort-keys '.' mounts_deleted.json) <(jq --sort-keys '.' mounts_current.json)

# // using th mount UID show correlating JSON portion which will need to be re-added
VUID=c2cbd2c9-33cd-dcf5-3cb3-3680cce6416a
jq -r ".entries[]|select(.uuid==\"${VUID}\")" mounts_deleted.json

The addition required onto the .entries portion of mounts should be made (via an editor) and the final document written back into Vault as well as reverified for it's validity before performing a final restart to all Vault instances that are running.

# // add back JSON blob for UID mount using any editor
nano mounts_current.json

# // Read / Assign updated JSON of mounts to VDATA
VDATA=$(jq -c '.' mounts_current.json)

# // Write new mounts definition back
vault write sys/raw/core/mounts value="${VDATA}"

# // verify write was successful by reading value & performing diff again

# // Restart is required for the updated mounts definition to be read
sudo systemctl restart vault

# // ensure all Vault instances are restarted and reading the edit mounts.

Once successfully restarted - a listing of the secrets mounts should list what was previously disabled which is now re-imported & re-enabled using the older snapshots.

vault secrets list -detailed
  # ...
  # // lists imported mount: c2cbd2c9-33cd-dcf5-3cb3-3680cce6416a

Resources

Articles in this section

See more

Related articles