How-to safely restart a Vault cluster running on Kubernetes – HashiCorp Help Center

Introduction

Expected Outcome

After reading this guide you will understand how to safely restart a Vault cluster running on Kubernetes with as minimal downtime possible.

Use Case

This guide can be used as a reference for those who are wishing to perform a restart of the Vault pods. This is useful for various scenarios, for example if a parameter in vault.hcl has been changed and a restart if required in order to pick up the changes.

This guide does not make use of the Kubernetes kubectl rollout restart command as we need to ensure both quorum is maintained and leader elections are kept to a minimum while being performed in a controlled fashion.

This guide does not intend to convey the process being outlined as the only possible method to achieve the desired outcome, instead it should be considered as one possible method.

---

The following considerations should be understood before completing this process:

This guide is written with Vault clusters using the Integrated Storage (raft) backend storage type in mind.
Vault will lose quorum and be unable to service client requests for a longer period of time if the kubectl rollout restart command is used. This can potentially result in a peers.json recovery being required.
PodDisruptionBudget can help minimise the amount of pods restarted simultaneously, however it cannot gracefully perform a leadership transfer for Vault or prevent the leader/active Vault pod from being terminated unexpectedly as part of the process, leading to unscheduled downtime.
A period of downtime / interruption to client requests is unavoidable as a leadership election must take place, during which Vault is unable to service client requests.
Following the process outlined in this document will result in the shortest possible interruption to service. This is achieved by ensuring the leadership transfer process only occurs once and the active/leader node is not restarted/cycled unexpectedly.
These steps do not detail the process of unsealing each pod, and instead assume an auto unseal / HSM seal type is in use.
The example commands used assume the Kubernetes namespace in which Vault is deployed into is also called vault. Adjust as necessary.

Procedure

In order to safely restart the Vault pods the following high level steps are completed:

Confirm the health of the cluster.
Identify and cycle/restart the follower pods.
Perform a leader election.
Restart the last pod.

Detailed Steps

Examine the output of vault operator raft autopilot state and ensure the following: [A]
1. The value for Healthy is true.
2. The count of voters matches your expectation, i.e. a 3 node Vault cluster should report 3 voters.
3. For each of the Servers entries:
  1. Node Status should be alive.
  2. Healthy should be true.
  3. Last Contact should be 5 seconds or less.
  4. Last Index should either be the same across all instances, or very close. A busy cluster with many regular writes is one scenario where the follower nodes may have a slightly different (but very close) value and still be considered healthy / in sync.
Examine the output of vault operator members and identify which pod is the Active Node. The pod which is the Active Node should be processed last. [B]
1. The other pods will show Active Node as false - these follower / standby pods must be restarted first.
Cycle/restart the first of the follower / standby pods, in this example being vault-0: kubectl -n vault delete po vault-0
Allow 45 seconds or so for a replacement pod to be started.
Examine the output of vault operator members to ensure the pod has rejoined the cluster. Last Echo should show the restarted pod is again communicating with the cluster by way of a less than 5 second time delta.
1. If the pod does not rejoin the cluster examine the output of kubectl -n vault describe po vault-0 and/or kubectl -n vault logs vault-0 to identify the source of the issue. Once the pod has rejoined the cluster the next steps can be completed.
Examine the output of vault operator raft autopilot state and ensure the restarted pod is healthy per the outputs we considered in step 1.
Repeat steps 3-6 for the remaining standby / follower pods, ensuring only one pod is cycled/restarted at a time.
Once all follower / standby pods have been cycled/restarted perform a leader election by running vault operator step-down. During the leader election process Vault will not service client requests.
Examine the output of vault operator members in order to see a new node is reported as the Active Node. In the example provided we can see the Active Node was initially vault-1, however after a leader election the Active Node is now vault-0. [C]
Cycle/restart the remaining pod which used to be the Active Node, in the example provided this is vault-1: kubectl -n vault delete po vault-1.
Repeat the review process outlined in steps 5 and 6 to ensure the last remaining pod has rejoined the Vault cluster.
Examining the output of kubectl -n vault get po should show all pods have a low Age value, confirming they have all been cycled/restarted. [D]

Additional Information

A. vault operator raft autopilot state -format=json | jq '.Servers[] | "\(.ID): \(.NodeStatus), Healthy: \(.Healthy), LastContact: \(.LastContact), LastIndex: \(.LastIndex)"' example output:

B. vault operator members example output prior to pod cycling/restart / leader election:

C. vault operator members example output after pod cycling/restarts & leader election:

D. kubectl -n vault get po example output after all pods have been cycled/restarted:

➜  ~ kubectl -n vault get po
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          9m
vault-1   1/1     Running   0          1m
vault-2   1/1     Running   0          5m