Re-initialize Consul Storage for Vault

Re-initializing Consul storage for Vault is a process of resetting the state of a Vault server's backend storage, which is maintained in Consul. This process is usually carried out when there is a need to start with a clean slate, or when there are inconsistencies or errors in the backend storage.

Re-initializing Consul storage for Vault is a critical process that should be approached with caution, as it involves deleting all of the data in the storage and starting over. This guide is intended to go through the process of re-initializing Vault with Consul as storage.

This process is done manually by following these steps:

1. Stop the Vault service on all Vault cluster nodes (Ex: systemctl stop vault)
2. Stop the Consul agent on all Vault nodes (Ex: systemctl stop consul)
3. Take a final snapshot from the Consul storage leader node to backup your data consul snapshot save backup.snap. Use consul snapshot save -stale backup.snap to create a potentially stale snapshot from any available server. This is useful for situations where a cluster is in a degraded state and no leader is available
4. Use the command consul kv delete -recurse -token=$CONSUL_TOKEN vault/to delete Vault data from Consul nodes, or clear the data folder on the Consul nodes. The path vault/depends on the storage path defined in the storage stanza in the Vault config file
5. Restart the Consul service on the Consul nodes(Optional) - (Ex: systemctl restart consul)
6. Start the Consul agents on the Vault nodes (Ex: systemctl start consul) - make sure to give the agents enough time to register.
7. Start the Vault service on the Vault nodes (Ex: systemctl start vault)

References:

https://developer.hashicorp.com/vault/tutorials/day-one-consul/deployment-guide

https://developer.hashicorp.com/vault/tutorials/day-one-consul/reference-architecture

https://developer.hashicorp.com/vault/docs/commands/operator/init