Introduction
This article documents a workaround for a known defect in Vault 1.11.3 & 1.12.x where authentication requests via the Kubernetes Auth backend intermittently fails.
A fix for this defect has been published in version 0.14.1 of the Kubernetes Auth method and was included in Vault 1.12.3, Vault 1.13.0 and later. The reference can be found in the Vault changelog entry:
- auth/kubernetes: Ensure a consistent TLS configuration for all k8s API requests (Pull requests #173 and GH-18716)
Problem
Vault authentication requests destined for the Kubernetes Auth backend intermittently fails returning a permission denied error.
Failing requests are mostly observed when made against a performance standby node while requests to the active node are successful.
Prerequisites (if applicable)
- Vault OSS 1.11.3, 1.12.0, 1.12.1 & 1.12.2
- Vault Enterprise 1.11.3, 1.12.0, 1.12.1 & 1.12.2
- Kubernetes Auth Method v0.14.0
Cause
Due to a known defect in the Vault versions listed above, https connections made from Vault to the Kubernetes cluster used for the token review process, may fail with a permission denied error.
The error printed in the Vault operational log for a failed request shows the following:
-
x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca") - Verifying the
kubernetes_ca_certparameter in the Kubernetes Auth method configuration shows a matching certificate authority with that of the Kubernetes cluster.
Overview of possible solutions
Solutions:
A reinitialization of the Kubernetes Auth method will trigger a reload of the stored Certificate Authority. This can be achieved by:
-
Reloading the Kubernetes Auth method with
vault plugin reload -plugin kubernetes -
Forcing a leader election with a step-down
vault operator step-down - Forcing a leader election by restarting the Vault leader node.
Outcome
All token review requests from Vault to Kubernetes should be successful after reloading the Kubernetes Auth method.
Additional Information
- Vault Documentation: Kubernetes Auth Method
- Vault Documentation: Plugin System