Recovery Using Recovery Mode in HashiCorp Vault

Avatar
Mehdi Ahmadi
  • Updated

Recovery mode in HashiCorp Vault is a specialized state that provides low-level access to Vault's internal storage. It is designed for exceptional circumstances where Vault's normal operations (such as starting the server or restoring snapshots) are not possible due to configuration issues or data corruption. This guide will walk you through using recovery mode safely and effectively.

 

Prerequisites

  1. Backup: Ensure you have a backup of your Vault storage before making any changes.

  2. Unseal or Recovery Keys: Ensure all required keys are accessible.

  3. Vault Status: Verify that the Vault instance requiring recovery is not active.

About Recovery Mode

Recovery mode enables direct interaction with Vault’s internal storage at the sys/raw/ path. It is limited to basic operations:

  • list: List entries under a given path.

  • read: View the contents of a specific entry.

  • delete: Remove an entry.

  • write: Update or create an entry.

Note: Recovery mode is intended for maintenance and recovery purposes only. No other Vault functionalities are available during this mode.

Starting Vault in Recovery Mode

  1. Stop Vault:

    systemctl stop vault
ps aux | grep vault

  2. Verify Configuration: Check how Vault is typically launched. For example:

    grep ExecStart /etc/systemd/system/vault.service
# Output: ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl

  3. Start in Recovery Mode: Use the -recovery flag:

    vault server -config=/etc/vault.d/vault.hcl -recovery

    Example output:

    ==> Vault server configuration:
Recovery Mode: true
Storage: raft
Version: Vault v1.7.3+ent.hsm

Generating a Recovery Token

A recovery token provides temporary access during recovery mode. Follow these steps to generate and use it:

  1. Initialize Token Generation:

    vault operator generate-root -recovery-token -init -format=json > rtoken.json
VOTP=$(jq -r '.otp' rtoken.json)
VNONCE=$(jq -r '.nonce' rtoken.json)

  2. Provide Unseal or Recovery Keys: Repeat for all keys:

    vault operator generate-root -recovery-token -nonce=${VNONCE}

  3. Decode the Recovery Token:

    VENCODED=<encoded_token>
vault operator generate-root -recovery-token -decode=${VENCODED} -otp=${VOTP}
export VAULT_TOKEN=<decoded_token>

Performing Recovery Tasks

1. Managing Resource Quotas

List Quotas:

vault list sys/raw/sys/quotas

Delete Quotas:

vault delete sys/raw/sys/quotas/config
vault delete sys/raw/sys/quotas/default_rate_limit_exempt_paths_toggle
vault delete sys/raw/sys/quotas/rate-limit/global

2. Removing Mounts (e.g., Secrets or Auth Engines)

Identify Mounts:

vault list sys/raw/core/mounts

Modify Mounts:

  1. Read the mounts configuration:

    vault read sys/raw/core/mounts

  2. Update the JSON document to exclude the unwanted mount:

    vault write sys/raw/core/mounts value='<updated_json>'

Delete Related Data:

vault list sys/raw/logical/<mount_guid>
vault delete sys/raw/logical/<mount_guid>/config
vault delete sys/raw/logical/<mount_guid>/role

3. Deleting Audit Devices

Remove Audit Configuration:

vault delete sys/raw/core/audit

Delete Audit Logs:

vault list sys/raw/audit/
vault delete sys/raw/audit/<path>

Stopping Recovery Mode

Once recovery tasks are complete, stop the recovery mode process and restart Vault normally:

  1. Stop Recovery Mode:

    pkill vault

  2. Restart Vault:

    systemctl start vault

  3. Verify: Check that Vault is functioning as expected and that the changes made are reflected.

Summary

By following these steps, you can use recovery mode effectively to resolve critical issues in HashiCorp Vault.

Additional Tips

  • Use the -format=json option with Vault CLI commands for easier scripting and parsing.

  • Always test recovery operations in a non-production environment before applying them in production.

 

Resources

Articles in this section

See more

Related articles