Vault Storage Backend Migration on Kubernetes, OpenShift, AKS or EKS

Avatar
Ashish Garg
  • Updated

Introduction

When migrating the Vault backend on Kubernetes, OpenShift, AKS or EKS there are additional steps because the pod is always running. When doing a backend migration, Vault needs to be stopped to complete this process.

Cause

Below are a list of example errors when attempting a backend migration, while Vault is deployed on Kubernetes, OpenShift, AKS or EKS. 

Example 1:

When migrating to Integrated Storage

2022-09-12T18:59:16.711Z [WARN] appending trailing forward slash to the path
Error migrating: error mounting 'storage_destination': failed to create fsm: failed to open bolt file: open /vault/data/vault.db: no such file or directory
  • This error means the path defined by the storage_destination path does not exist on the file system where the migration operation is taking place. Ensure the path defined in the migration configuration file is present on the file system. 

Example 2: 

Consul to Integrated Storage (Raft) Migration

2022-09-13T05:21:10.453Z [WARN]  appending trailing forward slash to the path
Error migrating: error mounting 'storage_destination': failed to create fsm: failed to open bolt file: timeout
  • This timeout error can occur if Vault is already running and/or initialized at the destination path. It is recommended to stop Vault during this process. Once the migration is completed, restart the pod. A separate path can be defined in the migration configuration file at the storage_destination path. After running the migration, the data can be moved to the desired path set in the Vault configuration and restarting the pod. 

Example 3:

Cluster already has state bootstrap error

Error migrating: error mounting 'storage_destination': could not bootstrap clustered storage: error bootstrapping cluster: cluster already has state
  • This error means that the storage at the destination path already has state or data. Delete or move the data for backup and re-run the migration process again.

Example 4: 

Integrated Storage (Raft) to S3 migration

Error migrating: error mounting 'storage_destination': unable to access bucket "vaultstorages3" in region "ap-south-1": InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 50VNJ39PP20YNDHJ, host id: u4+YSoMdDV7SQBy1NwN0k+8P/19qLSY2qkjzqqMxM0DLgrhNkdLVpFflwSMl1wiDLxpuv0l0Mdo=
  • The above error references issues with communicating to AWS S3. Verify all necessary parameters are present in thestorage_destinationstanza to communicate to the destination.

Overview

When running a storage migration while Vault is running on Kubernetes, OpenShift, AKS or EKS, Vault cannot be initialized at the destination. It is suggested to always take backups prior to running a storage migration. Additionally, test these operations in a lower environment when possible.

 

References

 

Articles in this section

Related articles