Introduction:
Restarting a Vault cluster is a crucial task to maintain the availability and integrity of secrets and encryption keys. This guide provides step-by-step instructions to help you restart your Vault cluster effectively when raft is the storage backend. It is recommended to schedule a maintenance window to minimize any potential disruptions during the restart.
Prerequisites:
Before restarting the Vault cluster, ensure you have access to your unseal key and root token (or admin token). It is also recommended to take a snapshot of the Vault cluster as a backup.
Step-by-Step Guide:
Step 1: Take a Snapshot (optional but recommended)
Take a snapshot of the active (preferred but not necessary) Vault cluster member using the command [1] :
vault operator raft snapshot save backup.snap
Step 2: Restart Standby Nodes
The steps below assumes you are using systemd.
1. Stop the Vault service on one of the standby nodes one at a time:
systemctl stop vault
2. Start the service on the standby node:
systemctl start vault
3. Unseal your node using your keys if auto-unseal is not setup.
4. Verify that the restarted node is added to the quorum [2] :
vault operator raft list-peers
5. Repeat these steps until all the standby nodes are restarted
Step 3: Step-Down and Change the Leader
1. Perform a step-down operation on the active node [3] :
vault operator step-down
2. Confirm the new leader node by checking the peers:
vault operator raft list-peers
The node marked as the "leader" is the new leader of the Vault cluster(should be one of the restarted one).
Step 4: Restart the Last Node
1. Restart the last node, which should now be in standby mode after the leader change instep 3. Follow the instructions shown in step 2.
2. Check the cluster status after the restart:
vault operator raft list-peers
Ensure that the all nodes have joined your cluster.
Conclusion:
By following these steps, you can successfully restart a Vault cluster while minimizing disruptions. Remember to schedule a maintenance window to ensure a smooth restart process. Keep in mind the importance of taking a snapshot for backup purposes.