Introduction
Problem
Kubernetes application pods are unable to authenticate to the Vault Kubernetes Auth method and permanently receive the following error:
403: permission denied
Prerequisites
- Vault -- all versions supporting the Kubernetes Auth Method
Cause
One possible cause of this failure is due to the fact that the Kubernetes auth method accesses the Kubernetes TokenReview API to validate if the provided JWT is still valid. Therefore, Service Account JWT tokens used in this auth method need to have appropriate access to the Kubernetes TokenReview API. The issue discussed in this particular article is the failure being caused by the Service Account JWT not having sufficient access to the Kubernetes TokenReview API and how to remedy the situation.
Should you wish to discover which Service Account is being used to access the Kubernetes TokenReview API firstly consult the Vault Audit logs and locate the request that is failing:
{"time":"2023-05-24T14:22:45.010515Z","type":"response","auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"id":"93afdcdc-b62a-a8c9-6de9-c5111d2f8d31","operation":"update","mount_type":"kubernetes","mount_accessor":"auth_kubernetes_bcb1dd98","namespace":{"id":"root"},"path":"auth/kubernetes/login","data":{"jwt":"hmac-sha256:9a6ddcbd5a93d8ddbe7e697234c7fac87b84743f2bc17938566fef8fe011623e","role":"hmac-sha256:706f91badce7ed968fb701a754ab9dd7b1ebc5a904c42d91b64c00dcf1963023"},"remote_address":"127.0.0.1","remote_port":57875},"response":{"mount_type":"kubernetes","mount_accessor":"auth_kubernetes_bcb1dd98"},"error":"permission denied"}
In the above example, the audit event contains a mount_accessor with a value of auth_kubernetes_bcb1dd98. This you can match up with the information in the Vault Operational logs:
2023-05-24T16:23:25.992+0200 [DEBUG] auth.kubernetes.auth_kubernetes_bcb1dd98: login unauthorized: err="{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"tokenreviews.authentication.k8s.io is forbidden: User \\\"system:serviceaccount:default:internal-app\\\" cannot create resource \\\"tokenreviews\\\" in API group \\\"authentication.k8s.io\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"group\":\"authentication.k8s.io\",\"kind\":\"tokenreviews\"},\"code\":403}"
Notice here that the Kubernetes Service Account being used is
system:serviceaccount:default:internal-app
Overview of possible solutions
Solution:
When configuring the Kubernetes Auth Method include this parameter token_reviewer_jwt and provide a Service Account JWT that has access to the Kubernetes TokenReview API to validate other JWTs during login. If this parameter is not set, the local Service Account token is used if running Vault in a Kubernetes pod or, if Vault is external to Kubernetes, the JWT submitted in the login payload will be used to access the Kubernetes TokenReview API.
Outcome
With the correct Service Account JWT provided to the token_reviewer_jwt parameter in the Kubernetes Auth method configuration, the authentication requests will succeed.
Additional Information
-
Note: when reading the configuration of the Kubernetes Auth method neither the presence of, nor the actual JWT token of the
token_reviewer_jwtparameter is returned in the output as it contains a secret. - Vault API: token_reviewer_jwt.
- Vault Documentation: Cluster Role Binding
- Vault Documentation: Kubernetes 1.24+
- Vault Documentation: Kubernetes Auth Method
- Vault Documentation: Kubernetes Auth Method API
- Vault Tutorial: Vault Agent with Kubernetes