How to Change the Vault Hostname on a Raft Cluster

Overview

This article will detail how to change the hostname on a primary or secondary Vault node operating in a Raft cluster. The details are specifically for changing the leader, but can be repurposed for any node in the cluster.

“Primary” will refer to the host being modified. “Secondary” the second host in the scenario and is applicable to whichever of the remaining two hosts picks up leadership. Leadership election in this case is done automatically using the Raft Consensus Protocol. The third host requires no actions in this scenario so is not referenced. “Leader” and “Follower” to be used for the hosts Raft state at that step in the instructions.

Prerequisites

•  A working Vault cluster with a Raft storage back-end composed of three hosts.

Procedure

• Stop the Vault service on the primary (leader) host. The remaining two secondaries will automatically elect a new Raft leader.

• Change the hostname on the primary.

• Update the Vault configuration file:

  • Change the Raft confignode_id& path and the Vault api_addr & cluster_addr to match the new hostname.

  • If any of these values are IP addresses instead of domain names, they can be skipped.

• Delete the old Raft storage on the primary.

• On the secondary node elected as leader, remove the old primary node ID with vault operator raft remove-peer <old domain name>.

• Start the Vault service on the primary. Raft will automatically recreate a new database.

• On the primary, join the Raft cluster as a follower using vault operator raft join https://<secondary hostname>:8200.

• Verify that the primary has rejoined the cluster with the new hostname and node ID with vault operator raft list-peers on both the primary and secondary.

• Wait for the Raft database to replicate on your primary. You can verify that this has been done by logging in with a token/method from the cluster and running a vault secrets list to make sure the displayed data matches that on the current leader. These steps ensure that both the token has replicated and that the secrets are up-to-date with the secondary (temporary leader).

• If you don’t want to promote the primary host back to leader, instructions end here. If you wish to promote the primary to leader again, continue with the instructions.

• Follow the steps in Recovering From a Permanently Lost Quorum in Raft Integrated Storage by creating a peers.json file on your secondary (temporary leader). The value for id should match the node_id from your primaries’ Vault config file, and address should match your cluster_addr and api_addr.

• Stop and start Vault on the secondary.

• Rerun vault operator raft list-peers on both primary and secondary to validate that the change has taken effect.

Additional Information

• Raft Integrated Storage