Recovery using recovery-mode

Avatar
Mehdi Ahmadi
  • Updated

Ensure you have a backup of the Vault storage before making any adjustments and that any entry being deleted or altered is known to you.

You may need to run Vault in recovery-mode in some exceptional circumstances.

Inadvertent changes in the configuration of Vault may have resulted in a state that's made it impossible to undo or interact with Vault; starting vault or restoring prior snapshots and other Vault CLI / API operations are also no longer possible.

The recovery-mode allows for direct low-level interaction with raw portions of the internal storage and it's limited to the operations: list, read, delete and write on the system root path of: sys/raw/...

During recovery-mode Vault does not provide for any functionality other than what's intended for maintenance and recovery purposes only.

To start Vault in recovery-mode you'll need to ensure that  the unseal or recovery keys (as applicable) are available and that the Vault needing recovery is not active or with a running process.

Vault should be launched using the existing configuration and with the additional CLI parameter of ... -recovery.

Once started, a non-persistent Recovery-Token will need to be generated for use as a VAULT_TOKEN during the period while in recovery-mode.

Demos

We'll start with a demonstration of deleting all resource-quotas whose properties are within the paths of: sys/raw/...

Other provided examples are of removing mounts (secrets or auth) as well as audit devices.

Starting Recovery-Mode & Generating A Recovery-Token

# // stop vault & ensure it's not running
systemctl stop vault
ps aux | grep vault

# // Verify how Vault is typically launched with what config:
grep ExecStart /etc/systemd/system/vault.service 
  # ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl

# // vault process & file permissions required (sudo / sudo -u)
# // start with: -recovery
vault server -config=/etc/vault.d/vault.hcl -recovery
  # ==> Vault server configuration:
  # .......................... 
  # Recovery Mode: true
  # Storage: raft
  # Version: Vault v1.7.3+ent.hsm
  # ..........................

# // on another terminal start recovery-token generation:
vault operator generate-root -recovery-token -init -format=json > rtoken.json
VOTP=$(jq -r '.otp' rtoken.json)
VNONCE=$(jq -r '.nonce' rtoken.json)

# // prepare unseal or recovery keys
vault operator generate-root -recovery-token -nonce=${VNONCE}
  # Operation nonce: ...
  # Unseal Key (will be hidden):
# // ^^^ REPEAT FOR ALL UNSEAL KEYS

# // IN THE END WRITE DOWN: encoded_token 
VENCODED=...encoded_token...

vault operator generate-root -recovery-token -decode=${VENCODED} -otp=${VOTP}
  # r.xyz...
# // ^ NOTE decoded VAULT_TOKEN
export VAULT_TOKEN=r.xyz...

vault list sys/raw/
  # ..........................

Deleting Resource-Quotas & Rate-Limits

# // List quotas configured 
vault list sys/raw/sys/quotas
  # ..........................
  # config
  # default_rate_limit_exempt_paths_toggle
  # rate-limit/

vault list sys/raw/index/pages/
  # ..........................

# // read all entries for any related property 
vault read sys/raw/index/pages/47 | grep rate-limit
  # !sys/quotas/rate-limit/global-d�$>�m�ېn�^�F��"�HMb�f�nbs

vault delete sys/raw/index/pages/47
vault delete sys/raw/sys/quotas/config
vault delete sys/raw/sys/quotas/default_rate_limit_exempt_paths_toggle
vault delete sys/raw/sys/quotas/rate-limit/...

vault list sys/raw/sys/
# // Confirm quotas are deleted.
# // Stop recovery mode. Restart normally now without any quotas.

You can continue to perform other tasks and once done the recovery-mode process should be stopped and Vault restarted in normal server mode to verify the changes made.

Disabling & Deleting Mounts

To disable a mount from being loaded it's internal UID reference is required to remove it from the JSON document at the path of sys/raw/core/mounts. The corresponding UID sub-paths in sys/raw/logical can also be further deleted to completely remove a mount secrets, roles and other definitions excluding any expired leases that are within sys/raw/sys/expire/....

Below is a demonstration of disabling a transit engine.

vault list sys/raw/logical
  # 8a8ea3d1-92b5-c17d-74f6-1d3aa5cbe6d1/
# // ^^ EG: this is Transform GUID since it has inner 'transformations` path

VGUID=8a8ea3d1-92b5-c17d-74f6-1d3aa5cbe6d1

# // UPDATE mounts document
vault read sys/raw/core/mounts

# // VDATA same JSON Value as above but excluding GUID of 'Transform' from the
# // mounts entries of: {"type":"mounts","entries":[ ... ]}
VDATA='{"type":"mounts","entries":[ ... ]}'

vault write sys/raw/core/mounts value="${VDATA}"

# // DELETE mounts logical path - first list all sub-path to gather list of docs
vault list sys/raw/logical/${VGUID}
vault list sys/raw/logical/${VGUID}/config/
vault list sys/raw/logical/${VGUID}/role/
vault list sys/raw/logical/${VGUID}/transformations
vault list sys/raw/logical/${VGUID}/transformations/fpe

# // ^^^ delete each listed document above:
vault delete sys/raw/logical/${VGUID}/config/upgrade
vault delete sys/raw/logical/${VGUID}/role/payments
vault delete sys/raw/logical/${VGUID}/transformations/fpe/card-number

vault list sys/raw/logical/${VGUID}
  # No value found at sys/raw/logical/...

# // Stop recovery mode. Restart normally where mount / engine is now removed.

Deleting Audit Devices

To disable all audit devices from being configured during boot delete the document at sys/raw/core/audit. Related audit properties are also detailed within sys/raw/audit/ which may also be deleted if prior audit references and salts are no longer required.

vault delete sys/raw/core/audit

vault list sys/raw/audit/

# // ^^ for each path:
vault list sys/raw/audit/.../

# // deleting documents in each sub-path eg:
vault delete sys/raw/audit/.../salt

Tips

When using the Vault CLI the parameter: -format=json can often help with listing items that need to be iterated; see: vault ... -help for more info.

Resources

Articles in this section

See more

Related articles