Introduction:
The Raft protocol has become a vital part of the Hashicorp ecosystem, most prominent example is the usage of the Raft protocol as a storage backend for Hashicorp Vault.
Expected Outcome:
This article will go over how to verify if the Raft peers that participate in a single Vault cluster is in sync and how to troubleshoot Raft cluster connectivity and storage issues.
Prerequisites:
- Use the Vault version that supports Raft integrated storage.
Use Case:
Let's say that your cluster is experiencing network issues or issues saving data to persistent storage, the first step to identify such a problem would be to compare the
Commit Index parameter numbers reported by the Raft peers.
Then numbers (indexes) reported by the Raft peers should be fairly close to one another.
Procedure:
-
Execute the
export VAULT_ADDR=https://127.0.0.1:8200; vault statuscommand on all Vault nodes which participate in the Raft integrated storage cluster. - Vault node 0 (IP: 10.10.42.200/24):
# export VAULT_ADDR=https://127.0.0.1:8200; vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.3+prem
Storage Type raft
Cluster Name PRIMARY
Cluster ID f6bb05a2-c62d-bcdf-a3e4-b269e764c997
HA Enabled true
HA Cluster https://10.10.42.200:8201
HA Mode active
Active Since 2021-10-04T08:18:27.782912057Z
Raft Committed Index 752
Raft Applied Index 752
Last WAL 69
- Vault node 1 (IP: 10.10.42.201/24):
# export VAULT_ADDR=https://127.0.0.1:8200; vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.3+prem
Storage Type raft
Cluster Name PRIMARY
Cluster ID f6bb05a2-c62d-bcdf-a3e4-b269e764c997
HA Enabled true
HA Cluster https://10.10.42.200:8201
HA Mode standby
Active Node Address https://10.10.42.200:8200
Performance Standby Node true
Performance Standby Last Remote WAL 69
Raft Committed Index 752
Raft Applied Index 752
- Vault node 2 (IP: 10.10.42.202/24):
# export VAULT_ADDR=https://127.0.0.1:8200; vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.8.3+prem
Storage Type raft
Cluster Name PRIMARY
Cluster ID f6bb05a2-c62d-bcdf-a3e4-b269e764c997
HA Enabled true
HA Cluster https://10.10.42.200:8201
HA Mode standby
Active Node Address https://10.10.42.200:8200
Performance Standby Node true
Performance Standby Last Remote WAL 69
Raft Committed Index 750
Raft Applied Index 750
The numbers (Indexes) in the
Raft Committed Index parameter signify if all the peers in the Raft cluster are in sync. The numbers (Indexes) should be fairly close between the Raft peers, having a small mismatch between the nodes is normal.
A slightly bigger gap between numbers (Indexes)
Raft Committed Index parameter might be observed when there is a high load on the cluster.
Keep in mind that this article does not take into account the
Performance standby feature of Enterprise version of Vault
Additional Information: