Expanding Vault Data Directory Storage Using Helm on Kubernetes

Introduction

Organizations running HashiCorp Vault on Kubernetes may need to expand the underlying storage size of the vault/data directory. Since Vault uses StatefulSets and PersistentVolumeClaims (PVCs), expanding storage requires a structured procedure to avoid data loss.
This KB outlines the recommended steps to safely scale up vault PVC storage when deployed using the Helm chart.

Prerequisites

Ensure you have the following before starting the storage expansion:

• Access to the Kubernetes cluster
• Vault unseal keys
• The original Vault Helm chart and values used for deployment
• Sufficient privileges to run Helm upgrade/uninstall/install commands
• A valid and tested Vault snapshot saved externally

Important Notes

1. PVC Retention Policy Must Be "Retain"

Before initiating the upgrade, verify that the StatefulSet PVC retention policy is set to Retain.

Run:

kubectl get statefulset vault -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'
Expected output:
{"whenDeleted":"Retain","whenScaled":"Retain"}

If this is not set to Retain, then STOP. Do not proceed becuase if its delete, statefulset will remove all the pvc.

2. StatefulSet Must Be Removed for Storage Changes

PVC size is an immutable Kubernetes field. Attempting to upgrade the StatefulSet without deleting it will fail with:

Error: UPGRADE FAILED: cannot patch "vault" with kind StatefulSet:
StatefulSet.apps "vault" is invalid:
spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 
'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
Deleting the StatefulSet is safe because the PVC retention policy protects your volumes.

3. Always Take a Vault Snapshot

Before starting the operation, take a snapshot and store it safely:

vault operator raft snapshot save snapshot.snap

4. Storage Class Must Support Expansion

Verify that your StorageClass allows volume expansion:

kubectl get sc <storage-class> -o jsonpath='{.allowVolumeExpansion}'
Expected output:
true

If not, storage expansion will not work.

5. This is an Offline process, so plan the maintenance window accordingly. 

Expand Storage Without Restoring From Snapshot (Keep One PVC)

This method preserves data by leaving one PVC intact, allowing Vault to re-join the cluster and replicate back to new nodes.

Step-by-Step Procedure

   Note:- It is recommended to take a snapshot prior to initiating the procedure, as the process temporarily results in vault-data-0 PV becoming a single point of failure

Step 1: Validate Current PVCs

Example current state (capacity = 15Gi):

kubectl get pv
kubectl get pvc

Ensure all pods and PVCs are healthy.

Step 2: Scale Down Vault StatefulSet

kubectl scale statefulset vault --replicas=0

Validate that PVCs still exist:

kubectl get pvc
All three PVCs (vault-0, vault-1, vault-2) should still be present.

Step 3: Delete PVCs for vault-1 and vault-2

⚠️ CRITICAL: Keep ONE PVC (data-vault-0) to avoid data loss.

kubectl delete pvc data-vault-1
kubectl delete pvc data-vault-2

You should now only see:

data-vault-0

Step 4: Uninstall Vault

helm uninstall vault

The StatefulSet and pods will be removed, but PVCs remain due to Retain policy.

Step 5: Update the Helm Values and Reinstall Vault

Modify values.yaml to increase storage, for example:

storage:
  size: 20Gi
Reinstall:

helm install vault hashicorp/vault -f values.yaml

Step 6: Validate New PVCs

New PVCs for vault-1 and vault-2 will be created with expanded size (e.g., 20Gi):

kubectl get pvc
You should see something like:

data-vault-0   15Gi
data-vault-1   20Gi
data-vault-2   20Gi

Step 7: Rotate Leader and Replace the Last PVC

 1:- Step down current Vault leader:

vault operator step-down

2:- Delete the PVC for vault-0:

First, remove finalizers:

kubectl edit pvc data-vault-0

Remove:

finalizers:
- kubernetes.io/pv-protection
Save and exit.

3:- Delete vault-0 pod:

kubectl delete pod vault-0

A new PVC with expanded storage will be created automatically.

Confirm:

kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-vault-0   Bound    pvc-e91a3ada-b45a-408e-bb80-1f555e013d91   20Gi       RWO            gp3-csi        <unset>                 7m38s
data-vault-1   Bound    pvc-caeeab6a-c22f-4cc3-9cfa-6011205d3d45   20Gi       RWO            gp3-csi        <unset>                 13m
data-vault-2   Bound    pvc-5a4074a4-8fee-4604-a248-0ed9415d81c9   20Gi       RWO            gp3-csi        <unset>                 9s

Now all PVCs should be at expanded size (e.g., 20Gi).