Problem
When attempting to upgrade a Vault standby node, the Vault operational logs display a looping error:
core: found new active node information, refreshing
sealwrap: unwrapping entry: key=core/leader/88f77ca8-7345-361a-e760-401ae5fa793b
core: found new active node information, refreshing
...
core: found new active node information, refreshing
sealwrap: unwrapping entry: key=core/leader/88f77ca8-7345-361a-e760-401ae5fa793b
Additionally, Vault returns a 500 error to client requests as it cannot determine cluster leadership.
Prerequisites
- Vault with Consul storage backend
Cause
- The error occurs when attempting an upgrade on a Vault standby node. This is because cluster leadership is lost and cannot be determined. The standby node then goes into a loop looking for a leader.
Overview of possible solutions
Solutions:
-
advertise_addr
- If being used, the value there needs to be configured with a unique IP address or DNS entry that only routes to a single instance of Vault.
-
cluster_addr and api_addr
- When these are not set, Vault will do its best to populate these values, although sometimes it can misconfigure the system. These values should be configured to be unique addresses specific to that instance of Vault.
- Ex. cluster_addr=<VM IP or DNS>:8201 and api_addr=<VM IP or DNS>:8200
Outcome
Vault will be able to establish cluster leadership and perform a successful upgrade.