Issue Summary
Error Message:
"Vault error: error performing token check: no lease entry found for token that ought to have one, possible eventual consistency issue."
Affected Environment:
- HCP Vault Dedicated Cluster (Tier: Can be any for multi-nodes cluster setup)
Root Cause
The error occurs due to an eventual consistency issue in token checks within the HCP Vault cluster. This issue can be seen in clusters that do not have performance replication enabled but still involve data replication between leader and follower nodes.
Explanation
The error stems from the replication mechanism where the leader node replicates its data to the follower nodes. In a typical 3-node HCP Vault cluster, the leader node is responsible for actively managing data, while the follower nodes act as performance standbys. The error indicates that a token check performed on a non-active node could not find the expected lease entry, likely due to the default allow_forwarding_via_token setting.
Resolution Steps
-
Feature Flag Activation:
- By default, the setting
allow_forwarding_via_tokenis unset in HCP Vault clusters. To mitigate this issue, it needs to be set tonew_token.
- By default, the setting
-
Action Required:
- Raise ticket with the support team to get this flag enabled. Refer this.
-
replication {
allow_forwarding_via_token = new_token
}
-
Expected Outcome:
- The update will propagate through the cluster, and you should stop seeing the "Vault error: error performing token check: no lease entry found for token that ought to have one, possible eventual consistency issue."
Follow-Up Actions
For Users:
- Monitor your HCP Vault cluster for any recurrence of the error.
- Report any persistent issues to the support team for further investigation.
Additional Information
Why the Error Occurs Without Performance Replication:
Even though performance replication is not enabled in your "Starter Small" tier, the leader node still replicates data to follower nodes as part of normal operation. This replication process is common to all multi-node HCP Vault clusters. The setting allow_forwarding_via_token is relevant to all requests sent to non-active nodes, preventing eventual consistency issues.
Documentation:
Refer to the HCP Vault configuration documentation for more details on the allow_forwarding_via_token setting and its implications on cluster operations.
Conclusion
The error "no lease entry found for token that ought to have one" is a known issue related to token checks in non-active nodes. With the activation of the feature flag and the update to your cluster, the issue should be resolved, ensuring smoother operation of your HCP Vault environment.
For further assistance or questions, please contact our support team.
Additional References:-
https://developer.hashicorp.com/vault/tutorials/enterprise/performance-standbys
https://developer.hashicorp.com/vault/docs/configuration/replication#allow_forwarding_via_token