Audience parameter usage with kube auth in vault (with caveats)

Introduction :

With usage of audience parameter in kubernetes auth role config, vault allows you to perform a audience based validation with kubernetes auth. This is an optional parameter, and making use of this parameter has direct relation with api-audience  parameter set on kube-apiserver level. 

Note that, there is common practise to make use of JWT auth for kubernetes using OIDC workflow for audience based validation on the JWTs. 

How it works:

When audience parameter is defined in kube auth role, audience in the logging JWT will be first matched against audience defined on the role within vault. If that succeeds, TokenReview request on kube api-server is performed, and authentication is allowed based on the review.

If the match fails, vault does not allow the auth attempt to go further for TokenReview and fails with below error :

Error writing data to auth/kubernetes/login: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* invalid audience (aud) claim: audience claim does not match any expected audience

A look at caveats with kubernetes auth method :

• Vault running on kubernetes cluster without token_reviewer_jwt:

In this configuration, Vault automatically utilizes the pod's service account token, which is associated with the API server's audience. Vault then uses this token to authenticate with the Kubernetes API server during the TokenReview process. For audience based validation to work, api-audiences needs to be set on api-server.

•  Vault running outside kubernetes cluster without token_reviewer_jwt :

In this case, the same token used for authentication is also intended for the TokenReview call. For audience based validation to work, api-audiences needs to be set on api-server.

• Vault running on kubernetes cluster with token_reviewer_jwt:

In this scenario, the token is manually created using the service account and is usually a long-lived token. If api-audiences is not set on api-server level, token_reviewer_jwt set on the kubernetes auth config must have default audience such as "https://kuberbetes.default.svc" for audience based validation to succeed.

• Vault running outside kubernetes cluster with token_reviewer_jwt :

Just like the previous scenario, the token is manually created using the service account and is usually a long-lived token. If api-audiences is not set on api-server level, token_reviewer_jwt set on the kubernetes auth config must have default audience such as "https://kuberbetes.default.svc" for audience based validation to succeed.

 How to troubleshoot issue while setting audience in kube auth role :

• Perform TokenReview API request outside vault (e.g. below) :

curl -kv -X "POST" "https://$K8S_ADDR:8443/apis/authentication.k8s.io/v1/tokenreviews" \
   -H 'Authorization: Bearer $TOKEN_REVIEWER_JWT' \
   -H 'Content-Type: application/json; charset=utf-8' \
   -d $'{
     "kind": "TokenReview",
     "apiVersion": "authentication.k8s.io/v1",
     "spec": {
        "token": "$SERVICE_ACCOUNT_JWT" 
             }
       }'

 TOKEN_REVIEWER_JWT is the JWT being used to perform TokenReview at kube api server.

SERVICE_ACCOUNT_JWT is the JWT attempting a vault login through kube auth method.

• Review kubernetes api-server logs if the TokenReview API request fails.

References :

Kubernetes auth method - Vault

Service account token volume projection - Kubernetes