How-to temporary force the leader election of the last node in a three nodes Vault cluster – HashiCorp Help Center

Introduction

Expected Outcome

In some edge cases there is a need to control the leader for a specific node and its eligibility to participate to RAFT storage Quorum. While there is no direct API available to perform this task, a change of the voting state could permit the leadership to be forced to a specific node using a step-down operation. This is achievable only where there are enough available nodes to maintain the Quorum.

Prerequisites (if applicable)

Vault Enterprise versions >1.12.x
Jq utility
Curl utility

Use Case

Temporary force the leader promote of a specific(last) node in a Vault cluster. For example, if there is a Vault cluster having 3 nodes as vault_2, vault_3, vault_4 configured with auto-seal mechanism, and there is a requirement to force the leader election to change to the last node of the cluster (vault_4), then currently there is no direct API or procedure to allow this activity.

As workaround there is an option, for a temporary time window, to force the first node to be in a non-voting state and perform a step-down operation. This will change the leader to the last node while preserving the cluster availability.

Special note: Running a RAFT with 2 nodes is considered a possible risk for a split-brain situation and lost quorum and an additional vault node should be temporary added to the cluster.

Procedure

Always perform a backup of your current Vault data and configuration

Check the current state of the nodes of the Vault cluster:

export VAULT_TOKEN=... # SET TOKEN
export VAULT_ADDR=... # SET ADDRESS
vault operator raft list-peers

The outputs may look like:

Node Address State Voter
---- ------- ----- -----
vault_2 127.0.0.1:8201 leader true
vault_3 127.0.0.1:8301 follower true
vault_4 127.0.0.1:8401 follower true

Perform a step-down operation and validate that the first node (vault_2) is not longer a voter node but still participating to the Vault cluster:

vault operator step-down && sleep 3 && vault operator raft list-peers

Likely output:

Success! Stepped down: http://127.0.0.1:8200

Node Address State Voter
---- ------- ----- -----
vault_2 127.0.0.1:8201 follower true
vault_3 127.0.0.1:8301 leader true
vault_4 127.0.0.1:8401 follower true

Validate that there are enough nodes to fulfil the quorum (or Min Quorum = 0):
- ```
vault operator raft autopilot get-config
```

The output might may look like:

Key Value
--- -----
Cleanup Dead Servers false
Last Contact Threshold 10s
Dead Server Last Contact Threshold 24h0m0s
Server Stabilization Time 10s
Min Quorum 0
Max Trailing Logs 1000
Disable Upgrade Migration false

Add the autopilot_upgrade_version = "1.18.0.1" to the raft storage stanza:

storage "raft" {
  path = "/opt/vault/data/raft-vault_2/"
  node_id = "vault_2"
  autopilot_upgrade_version = "1.18.0.1"
}

Restart the Vault service or process that is servicing the first node (vault_2)

Inspect the autopilot state by executing a GET operation on the API path:/v1/sys/storage/raft/autopilot/state- eg:

curl -s -H "X-Vault-Token: ${VAULT_TOKEN}" \
  $VAULT_ADDR/v1/sys/storage/raft/autopilot/state \
  | jq -r ".data.upgrade_info"

Output may resemble:

{
  "other_version_voters": [
    "vault_3",
    "vault_4"
  ],
  "status": "await-new-voters",
  "target_version": "1.18.0.1",
  "target_version_non_voters": [
    "vault_2"
  ]
}

Check the state of the first node and the leadership node of the Vault cluster:
- ```
vault operator raft list-peers
```

Output:

Node Address State Voter
---- ------- ----- -----
vault_2 127.0.0.1:8201 follower false
vault_3 127.0.0.1:8301 leader true
vault_4 127.0.0.1:8401 follower true

Execute a step-down operation and examine the vault peers
- ```
vault operator raft step-down
```

Check the state of the first node and the leadership node of the Vault cluster:

```
vault operator raft list-peers
```

Node Address State Voter
---- ------- ----- -----
vault_2 127.0.0.1:8201 follower false
vault_3 127.0.0.1:8301 follower true
vault_4 127.0.0.1:8401 leader true

Remove the autopilot_upgrade_version = "1.18.0.1" in the raft storage stanza:

storage "raft" {
  path = "/opt/vault/data/raft-vault_2/"
  node_id = "vault_2"
}

Restart the Vault service or process that is servicing the first node (vault_2)

(Optional) Execute a step-down for changing the leadership to first node:

vault operator step-down && sleep 3 && vault operator raft list-peers

The outputs may look like:

Address State Voter
---- ------- ----- -----
vault_2 127.0.0.1:8201 leader true
vault_3 127.0.0.1:8301 follower true
vault_4 127.0.0.1:8401 follower true

Additional Information

Vault Concept - Storage Backups
Vault Tutorials - RAFT Auto-upgrade-automation
RAFT Architecture - Consensus Protocol