Vault Storage Backend Migration

Avatar
Ashish Garg
  • Updated

Introduction

We have seen many customers using different storage backends with the Vault such as s3, integrated storage i.e. raft, consul, MySQL, dynamo DB, etc on their on-prem setup. But the challenges come when someone using the OSS version of the Vault in their environment and then wants to shift to the enterprise version of the Vault to use the enterprise feature such as DR/PR etc, at that time they have to migrate their data on a new cluster with the enterprise version without any issue. At that time, they can use the vault operator migrateutility.

 

Common Issues

We have seen below common issues that generally face by the customers:-

  1. How to write backend migration configuration file, in our document it refers with name migrate.hcl
  2.  Which parameters do we have to include in our stanzas in the migration configuration file i.e. migrate.hcl 
  3. Where to run that migration configuration file i.e. migrate.hcl either on the new cluster or on the old cluster of the Vault.
  4. How to check whether the migration got completed or not & what to do in case of interruption at the time of migration is running

Solution

So for performing the migration we have a pre-requisite that, takes a snapshot of the current backend & then the vault process should be stopped to maintain the data consistency and the destination cluster should not be initialized prior to migrating operation. If the process of the Vault is running then the migration utility will not be able to overwrite the data to the destination storage and this operation doesn't change anything on source data.

 

Sol 1) For writing the configuration file we have to define the stanzas storage_source and storage_destination in any order in the configuration file i.e. migrate.hcl, refer below example:-

root@vaults0:/home/vagrant# cat migrate.hcl
storage_source "NameOfTheSourceStorage" {
parametersOfTheSourceStorage
}

storage_destination "NameofTheDestinationStorage" {
parametersOfTheDestinationStorage
}

Sol 2) This is the confusing part where most people make the mistake, so for getting the parameter for stanzas storage_source and storage_destination we should visit this storage link & click on the desired storage option from the left of the link, once the page comes up copy the content mentioned in the default storage stanza and put that content inside the stanzas storage_source and storage_destination, sometimes we have to add the additional parameter that we get from the specific storage link page where detailed information about the parameter is present so we have to use it according to our use case.

Sol 3) This is the tricky part where we have to run the migration command vault operator migrate -config=migrate.hcl , so I have a below couple of cases for the same:-

  • In case of file to raft migration, we have to run the migration command on source i.e on vault with file backend & also need to create a path that is mentioned in storage_destination for a raft on the vault server with file backend, and then run the migration & it will copy the data to raft DB by creating the integrated storage at the path created on vault with file backend & then copy those raft from file server & move it to destination cluster with raft storage.
  • In case of s3 to raft & raft to s3 migration, always run the migration operation on the vault with raft cluster, otherwise, you will get the no such file or directory error.
  • In case of consul to raft & raft to consul  migration, run the migration operation on the vault with the raft cluster           
**Important Note on where to run the migration operation**

   1) We should see the storage type & its parameter and then need to decide where to 
      run the migration operation, so if both stanzas i.e. storage_source & 
      storage_destination contain the parameter that uses filesystem i.e. path on the 
      server then we have to run the migration on storage_source & then copy the migrated 
      data from the path defined in storage_destination on storage_source server to destination 
      vault cluster.

   2) If any storage stanza contains the URL and there is no parameter present that 
      contains the path or filesystem then we have to run the migration operation on 
      another storage stanza where the path or filesystem is defined. 

   3) If both storage stanza contains the URL but there is no parameter present that 
      contains the path or filesystem in both storage stanza, then we can run the 
      migration operation on any storage stanza's cluster.

 

Sol 4) So for checking the migration process completion we have to look for string, Success! All of the keys have been migrated. , once we will get this message means migration has been completed successfully.  In case of interruption, we can resume the migration by running the command vault operator migrate -config=migrate.hcl -start="core/auth". In -start, we have to pass the migration starting key prefix. Only keys at or after this value will be copied

 

Below is some migration.hcl file snippet:-


#S3 to Raft migration configuration snippet
storage_source "s3" {
access_key = ""
secret_key = ""
bucket = ""
region = ""
session_token = ""
}

storage_destination "raft" {
path = "/path/to/the/raft"
node_id = "node_name"
}
cluster_addr = "http://X.X.X.X:8201"

======================================================================================

#File to Raft migration configuration snippet
storage_source "file" {
path = "/path/to/file/"
}

storage_destination "raft" {
path = "/path/to/raft/"
node_id = "node_name"
}
cluster_addr = "https://X.X.X.X:8201"

======================================================================================

#Consul to Raft migration configuration snippet
storage_source "consul" {
address = "X.X.X.X:8500"
path = "/path/to/vault/key"
}

storage_destination "raft" {
path = "/path/to/raft/"
node_id = "node_name"
}
cluster_addr = "https://X.X.X.X:8201"

======================================================================================

#S3 to S3 migration configuration snippet
storage_source "s3" {
access_key = "ASIA5UFMDRN5KXNVSNHX"
secret_key = "Tj63l0BlAn8g9IW1i/eMd+6S0zcKRlgLB+GC/lPc"
bucket = "vaultstorages3"
region = "ap-south-1"
session_token = "" #Use this if required
}

storage_destination "s3" {
access_key = "ASIA5UFMDRN5KXNVSNHX"
secret_key = "Tj63l0BlAn8g9IW1i/eMd+6S0zcKRlgLB+GC/lPc"
bucket = "vaultstorages3dest"
region = "ap-south-1"
session_token = "" #Use this if required
}

======================================================================================

 

References

https://www.vaultproject.io/docs/v1.9.x/commands/operator/migrate

https://learn.hashicorp.com/tutorials/vault/raft-migration

https://www.vaultproject.io/docs/configuration/storage

https://support.hashicorp.com/hc/en-us/articles/9624215940883-Vault-Storage-Backend-Migration-on-Kubernetes-OpenShift-AKS-and-EKS -- For storage migration related errors & how to perform migration on Kubernetes

Perform the test in the local lab & then put the scenarios on the basis of it.

 

 

 

 

 

 

 

 

 

 

Articles in this section

Related articles