This article demonstrates and explains how to set up Vault on Kubernetes using Helm chart which includes SealHA.
Seal HA provides the means to configure at least two auto-unseals (no more than three) to have resilience against an outage of a seal service or mechanism.
Using Seal HA involves configuring extra seals in Vault's server configuration file and restarting Vault after having enabled the Seal HA beta feature by setting the below environment variable:
VAULT_ENABLE_SEAL_HA_BETA=true
Before using Seal HA, one must have upgraded to Vault 1.15 or higher and started with the environment variable and a single seal.
Warning: At the time of writing this article, this feature is available as a Beta for evaluation and should not be used in production deployments of Vault.
Pre-requisites
The article is tested on Apple MacOS 13.6.1 with the following:
- Docker v24.0.7
- Minikube v1.31.1
- Kubernetes version v1.27.3
- Helm v3.12.2
- Helm Chart v0.27.0 that has Vault v1.15.2+ent
Setup
- Start a local Kubernetes cluster with one node:
~ % minikube start --cpus='4' --disk-size='100000mb' --memory='4g'
~ % kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 10m v1.27.3
- Deploy 3 node Vault cluster with Integrated Storage & primary auto-unseal mechanism (here we're using AWS KMS) using the helm chart with a minimum of these details in a file named "VaultCluster.yaml" along with the Environment Variable needed for SealHA which is a pre-requisite to enable the feature:
# Add the HashiCorp Helm repository.
~ % helm repo add hashicorp https://helm.releases.hashicorp.com
# Update all the repositories to ensure helm is aware of the latest versions.
~ % helm repo update
# Install the latest version of the Vault Helm chart with Integrated Storage.
~ % kubectl create secret generic vault-license --from-file=$ABSOLUTE_PATH_TO_LICENSE_FILE
~ % helm install vault hashicorp/vault --values VaultCluster.yaml
~ % k get pods --all-namespaces | grep -i vault
NAMESPACE NAME READY STATUS RESTARTS AGE
default vault-0 1/1 Running 0 4m58s
default vault-1 1/1 Running 0 4m58s
default vault-2 1/1 Running 0 4m58s
~ % kubectl exec vault-0 -- vault operator init -recovery-shares=1 -recovery-threshold=1 -format=json > cluster-keys.json
~ % VAULT_UNSEAL_KEY=$(jq -r ".recovery_keys_b64[]" cluster-keys.json)
~ % kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
~ % kubectl exec -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY
~ % kubectl exec -ti vault-2 -- vault operator raft join http://vault-0.vault-internal:8200
~ % kubectl exec -ti vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY
~ % helm list -a
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
vault default 1 2024-01-26 19:22:00.551165 +0530 IST deployed vault-0.27.0 1.15.2
- If you would describe any pod, you must be able to see the Environment Variable as well, such as:
~ % kubectl describe pod vault-0
.
.
.
Environment:
.
VAULT_ENABLE_SEAL_HA_BETA: true
.
.....
Upgrade to Seal HA
- Once the Vault Cluster is up and running, add another new AWS seal stanza, and upgrade the cluster using helm. Here is the reference of the values file with updated details:
~ % helm upgrade vault hashicorp/vault -f UpgradeVaultCluster.yaml
# The value of Revision will increase by 1 point if the upgrade is successful.
~ % helm list -a
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
vault default 2 2024-01-26 19:42:00.651155 +0530 IST deployed vault-0.27.0 1.15.2
- Please note, that even though the helm deployment is upgraded, the Vault pods are yet to know the new settings that are pushed during the helm upgrade, and to do that, we'll have to perform a rolling restart of the Vault pods, and to do that, run:
~ % kubectl delete pod vault-0 -n default #wait for the pod/container to come up with Vault service [Ready - 1/1]
~ % kubectl delete pod vault-1 -n default #wait for the pod/container to come up with Vault service [Ready - 1/1]
~ % kubectl delete pod vault-2 -n default
- You can also opt for another native rolling restart method based on how have you deployed the Vault on Kubernetes like:
~ % kubectl rollout restart deployment <deployment_name> -n <namespace>
'or'
~ % kubectl rollout restart pod -l app=vault
#where 'app=vault' is a label sticked to the pods
- Once the pods are restarted or redeployed, you'll be able to see "Seal Type - - multiseal" when running 'vault status' on any pod. Such as:
~ % vault status
Key Value
--- -----
Seal Type multiseal
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 1
Threshold 1
Version 1.15.4+ent
Build Date 2023-12-05T01:50:21Z
Storage Type raft
------
Rollback
To roll back to the previous state with one seal if needed, all you need to do is to comment out or remove the second seal stanza and priority parameter from the first seal stanza and then do the helm upgrade again followed by a rolling restart of Vault pods.