Introduction
Problem
Joining a new Vault node to an existing Vault Cluster which already has Seal High Availability enabled fails with:
Dec 04 20:32:41 joining-node vault[7028]: 2024-12-04T20:32:41.872+0200 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.0.37:8200
Dec 04 20:32:41 joining-node vault[7028]: error=
Dec 04 20:32:41 joining-node vault[7028]: | error during raft bootstrap init call: Error making API request.
Dec 04 20:32:41 joining-node vault[7028]: |
Dec 04 20:32:41 joining-node vault[7028]: | URL: PUT https://192.168.0.37:8200/v1/sys/storage/raft/bootstrap/challenge
Dec 04 20:32:41 joining-node vault[7028]: | Code: 503. Errors:
Dec 04 20:32:41 joining-node vault[7028]: |
Dec 04 20:32:41 joining-node vault[7028]: | * Vault is sealed
Dec 04 20:32:41 joining-node vault[7028]:
Dec 04 20:32:41 joining-node vault[7028]: 2024-12-04T20:32:41.873+0200 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.0.1:8200 error="error decoding raft bootstrap challenge: proto:\u00a0cannot parse invalid wire-format data"
Dec 04 20:32:41 joining-node vault[7028]: 2024-12-04T20:32:41.874+0200 [ERROR] core: failed to get raft challenge: leader_addr=https://192.168.0.2:8200 error="error decoding raft bootstrap challenge: proto:\u00a0cannot parse invalid wire-format data"
In this scenario, the joining node doesn't have Seal High Availability enabled, which is expected as Seal High Availability should only be enabled on newly added Vault nodes after the respective nodes have successfully joined an existing Vault cluster.
Prerequisites (if applicable)
- Vault Enterprise
- Seal High Availability
Cause
-
This is a known issue and is planned to be resolved in a future version of Vault
-
The issue is present in Vault versions up to 1.16.13, 1.17.9, 1.18.2
Overview of possible solutions (if applicable)
Solutions:
- This issue has been in addressed in Vault versions 1.16.14, 1.17.10, 1.18.3 and newer
- For earlier version of Vault the only known workaround is to disable Seal High Availability for the entire Vault cluster, prior to adding a new Vault node to the Vault Cluster.
Outcome
After disabling Seal High Availability for the entire Vault cluster, it is possible to add a new Vault node to the Vault cluster.