Introduction
This article documents a workaround for a known defect in Vault 1.11.3 & 1.12.x where authentication requests via the Kubernetes Auth backend intermittently fails.
The defect is already logged and a fix is anticipated in a future release which will be highlighted here whenever that's made.
Problem
Vault authentication requests destined for the Kubernetes Auth backend intermittently fails returning a permission denied error.
Failing requests are mostly observed when made against a performance standby node while requests to the active node are successful.
Prerequisites (if applicable)
- Vault OSS 1.11.3, 1.12.0, 1.12.1 & 1.12.2
- Vault Enterprise 1.11.3, 1.12.0, 1.12.1 & 1.12.2
- Kubernetes Auth Method v0.14.0
Cause
Due to a known defect in the Vault versions listed above, https connections made from Vault to the Kubernetes cluster used for the token review process, may fail with a permission denied error.
The error printed in the Vault operational log for a failed request shows the following:
-
x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca") - Verifying the
kubernetes_ca_certparameter in the Kubernetes Auth method configuration shows a matching certificate authority with that of the Kubernetes cluster.
Overview of possible solutions
Solutions:
A reinitialization of the Kubernetes Auth method will trigger a reload of the stored Certificate Authority. This can be achieved by:
-
Reloading the Kubernetes Auth method with
vault plugin reload -plugin kubernetes -
Forcing a leader election with a step-down
vault operator step-down - Forcing a leader election by restarting the Vault leader node.
Outcome
All token review requests from Vault to Kubernetes should be successful after reloading the Kubernetes Auth method.
Additional Information
- Vault Documentation: Kubernetes Auth Method
- Vault Documentation: Plugin System