Vault CLI Guide to Disaster Recovery Replication Failover

Avatar
Mallika Dasanoor
  • Updated

Overview

This guide provides CLI commands and API links for a basic Disaster Recovery replication failover. This guide focuses on CLI commands for Vault versions 0.9.0 and newer, which include generating a Disaster Recovery Operation Token.

If you are on an older version, it is highly recommended to upgrade to take advantage of replication-related bug fixes and feature enhancements. See changelog and Upgrade Guides.

Assumptions

This guide assumes that you already have Disaster Recovery replication enabled and functioning between a primary and secondary cluster. It also assumes that the clusters are in a healthy "stream-wals" state.

The original Primary is referred to as "Server A" and the original Secondary "Server B".

Disaster Recovery Failover for versions 0.9.0 and newer

Step 1: Demote DR Primary (Server A) with:

vault write -f sys/replication/dr/primary/demote

Step 2: Generate a DR Operation Token on Server B, to use for promotion. For example:

vault operator generate-root -dr-token -generate-otp

vault operator generate-root -dr-token -init -otp=OTP_HERE

Run the following update call multiple times, using a different unseal (or recovery) key each time, until your unseal/recovery threshold is met and it returns an encoded DR Operation Token

curl \
    --request PUT \
    --data '{"key":"UNSEAL_OR_RECOVERY_KEY_HERE", "nonce":"NONCE_HERE"}' \
    VAULT_ADDR_OF_SERVER_B/v1/sys/replication/dr/secondary/generate-operation-token/update

Then, decode the encoded DR operation token that is returned

vault operator generate-root -dr-token -otp="OTP_HERE" -decode="ENCODED_TOKEN_HERE"

Step 3: Promote Server B to Primary using the DR Operation Token you just generated:

vault write sys/replication/dr/secondary/promote dr_operation_token=DR_OP_TOKEN_HERE

Step 4: Generate a secondary activation token on Server B so that you can activate Server A as its secondary:

vault write sys/replication/dr/primary/secondary-token id=CREATE_AN_ID_HERE

Note: ID can be any name you would like to give to the token

Step 5: Generate a DR Operation Token on Server A to use as part of its update-primary call. Use the same steps used above for step 2, only this time do it on Server A.

vault operator generate-root -dr-token -generate-otp

vault operator generate-root -dr-token -init -otp=OTP_HERE

Run the following update call multiple times, using a different unseal (or recovery) key each time, until your unseal/recovery threshold is met, and it returns an encoded DR Operation Token

curl --request PUT --data '{"key":"UNSEAL_OR_RECOVERY_KEY_HERE", "nonce":"NONCE_HERE"}' VAULT_ADDR_OF_SERVER_A/v1/sys/replication/dr/secondary/generate-operation-token/update

Then, decode the encoded DR operation token that is returned

vault operator generate-root -dr-token -otp="OTP_HERE" -decode="ENCODED_TOKEN_HERE"

Step 6: Update Server A to see Server B as its new primary, using the update-primary endpoint:

vault write sys/replication/dr/secondary/update-primary dr_operation_token=DR_OP_TOKEN_HERE token=SECONDARY_ACTIVATION_TOKEN_HERE

Step 7: Now, when you run vault read sys/replication/dr/status on each server, you should see Server B as primary with the ID of Server A as its known secondary. Likewise, you should see Server A as secondary with Server B's cluster address as its primary cluster address.

You'll know they're fully synced when the last_wal on the Primary matches the last_remote_wal on the Secondary.

Resources

Articles in this section

See more

Related articles