How To Setup Readiness/Liveness Probes with Replication

This article will discuss how to configure and enable readiness and liveness probes for Vault running in Kubernetes and with replication.

Scenario

Customers may have readiness and liveness probes enabled before enabling replication.

If the customer already has probes setup and enables replication, the pods on the secondary will fail the health checks with a 503 since Vault will seal on the secondary when replication is enabled.

Recommendation

1. Update Helm chart and set readiness and liveness probes to false on the secondary cluster
   

   1. readinessProbe:
        enabled: false
        path: '/v1/sys/health?standbyok=true&perfstandbyok=true'
      livenessProbe:
        enabled: false
        path: '/v1/sys/health?standbyok=true&perfstandbyok=true'

2. Deploy new version of Helm chart for the secondary cluster
   

   1. helm upgrade vault hashicorp/vault -f <vault-values>.yaml

3. Reschedule pods on the secondary cluster

   1. # Run for each pod in the cluster starting with the standbys before moving to the active
      kubectl delete pod <vault-pod>

4. Unseal pods on the secondary using secondary unseal keys. This process is automatic if auto-unseal is configured.

   1. kubectl exec -ti <vault-pod> -- vault operator unseal $UNSEAL_KEY

5. Enable replication (PR or DR)
   
   1.  On the primary cluster
      

      1. vault login <token>
         kubectl exec -ti <vault-pod> -- vault write -f sys/replication/<replication_type>/primary/enable
         kubectl exec -ti <vault-pod> -- vault write sys/replication/<replication_type>/primary/secondary-token id="secondary" -format=json 

   2. On the secondary cluster

      1. vault login <token>
         kubectl exec -ti <vault-pod> -- vault write sys/replication/<replication_type>/secondary/enable token=$TOKEN

6. Update Helm chart and set readiness and liveness probes to true on the secondary. Note: the below path will need to be modified for your environment. Please refer to our sys/health doc for more details
   

   1. readinessProbe:
        enabled: true
        path: '/v1/sys/health?standbyok=true&perfstandbyok=true'
      livenessProbe:
        enabled: true
        path: '/v1/sys/health?standbyok=true&perfstandbyok=true'

7. Deploy new version of Helm chart for the secondary cluster
   

   1. helm upgrade vault hashicorp/vault -f <vault-values>.yaml 

8. Reschedule pods on the secondary cluster
   

   1. # Run for each pod in the cluster starting with the standbys before moving to the active
      kubectl delete pod <vault-pod>

9. Unseal pods on the secondary using primary cluster unseal keys
   

   1. kubectl exec -ti <vault-pod> -- vault operator unseal $UNSEAL_KEY

Additional Information

• Liveness/Readiness Probes

• Vault Helm
• Replication
• /sys/health API