Kubernetes auth method : Permission Denied error

Avatar
Mallika Dasanoor
  • Updated

This article outlines the configuration needed to help fix the permission denied error in k8s auth

Introduction

The kubernetes auth method can be used to authenticate with Vault using a Kubernetes Service Account Token. This method of authentication makes it easy to introduce a Vault token into a Kubernetes Pod.

Problem

When a pod attempts to authenticate to vault via the configured kubernetes auth method; it gets Permission Denied error

Cause

  • When a pod tried to authenticate to vault and you see an error like this in the audit logs of the vault pod: 
    {"time":"2020-05-23T00:16:35.5749466Z","type":"response","auth":{"token_type":"default"},"request":{"id":"6f594d4c-40b2-dbfc-d4ba-ef53d1ba03f9","operation":"update","mount_type":"kubernetes","namespace":{"id":"root"},"path":"auth/k8s-auth/login","data":{"jwt":"---BLAH---","role":"test"},"remote_address":"10.121.22.67"},"response":{"mount_type":"kubernetes"},"error":"permission denied"}

This error message is usually caused by one of five reasons:

  • The JWT used for authenticating to vault does not have the right role associated to it specified during the login

  • The JWT does not belong to the right service account

  • The role in k8s auth is not configured properly

  • The service account configured with k8s auth does not have the right cluster role binding associated with it
  • The Vault policy associated with the role does not have the permissions to the path being requested on the Vault side

Overview of possible solutions (if applicable)

Solutions:

Decode your JWT being used for login using jwt.io and examine the payload data to make sure the service account and namespace look right

Below is an example configuration of a working k8s auth.  In this below example:

  • vault-auth is the service account with which k8s auth is configured.

  • test-cloud is the service account used to login to vault using the k8s auth method

  • test is the namespace of the service account test-cloud

  • role-tokenreview-binding is the name of the cluster role binding the service accounts (vault-auth) needs to be associated with

# Create k8s namespace test

$ kubectl create namespace test
namespace/test created

# Create a service account, secret and ClusterRoleBinding with the necessary permissions to allow the service account test-cloud to perform token reviews with k8s 

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-cloud
  namespace: test
---
apiVersion: v1
kind: Secret
metadata:
 name: test-cloud
 namespace: test
 annotations:
   kubernetes.io/service-account.name: test-cloud
type: kubernetes.io/service-account-token
EOF
serviceaccount/test-cloud created
secret/test-cloud created

# Create a service account, secret and ClusterRoleBinding with the necessary permissions to allow vault to perform token reviews with k8s (Make note that clusterRoleBinding has been created associating serviceaccount  vault-auth)

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default
EOF
serviceaccount/vault-auth created
secret/vault-auth created
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

# Enable k8s auth method in vault

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

# Get the JSON web token (JWT) for vault-auth service account in default namespace to be used by vault k8s config

$ TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode)

# Retrieve the k8s CA certificate

$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)

# Retrieve the k8s host URL

$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

# Configure the k8s auth method to use the above sa token, location of the k8s host and its certificate from above

$ vault write auth/kubernetes/config   token_reviewer_jwt="$TOKEN_REVIEW_JWT"   kubernetes_host="$KUBE_HOST"   kubernetes_ca_cert="$KUBE_CA_CERT" disable_local_ca_jwt="true"
Success! Data written to: auth/kubernetes/config

# Read the k8s config

$ vault read auth/kubernetes/config
Key                       Value
---                       -----
disable_iss_validation    false
disable_local_ca_jwt      true
issuer                    n/a
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
MIIC5zCCAc+gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTIxMDYyMDE0MzA1OVoXDTMxMDYxOTE0MzA1OVowFTETMBEGA1UE
AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALZV
jjkUtgyFX/rodABD7SqRPuygBpSik4u2wHLm5Wr101+1aZ+nO7L63ykqAbOL89Nm
-----END CERTIFICATE-----
kubernetes_host           https://172.17.0.61:8443
pem_keys                  []

# Write a policy to associate with the devweb-app role used for login by test-cloud service account

vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read"]
}
EOF

Success! Uploaded policy: devwebapp

# Associate the role to the serviceaccount and the policy:

vault write auth/kubernetes/role/devweb-app \
  bound_service_account_names=test-cloud \
  bound_service_account_namespaces=test \
  policies=devwebapp \
  ttl=24h
Success! Data written to: auth/kubernetes/role/devweb-app

# Retrieve the JWT for test-cloud sa to login

TOKEN_REVIEW_SJWT=$(kubectl get secret  test-cloud -n test -o go-template='{{ .data.token }}' | base64 --decode)

# Perform a k8s login using test-cloud serviceaccount's JWT and the role

$  curl \
>     --request POST \
>     --data '{"jwt": "'$TOKEN_REVIEW_SJWT'", "role": "devweb-app"}' \
>     http://127.0.0.1:8200/v1/auth/kubernetes/login
{"request_id":"80cac595-c206-348f-2da5-29fe029eddd7","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.mBUJCVNNFKtUVuWVVgfPi9Sj","accessor":"4NoXtX7F9bcu19RZ7TAbtcjT","policies":["default","devwebapp"],"token_policies":["default","devwebapp"],"metadata":{"role":"devweb-app","service_account_name":"test-cloud","service_account_namespace":"test","service_account_secret_name":"test-cloud","service_account_uid":"6d475e37-0288-4070-ad8a-e44abc87b496"},"lease_duration":86400,"renewable":true,"entity_id":"841af42c-87ff-8441-b2d5-0c02ea062384","token_type":"service","orphan":true}}

The login is successful.

If the login is successful, that indicates everything is set up right.

Additional Information

Articles in this section

See more

Related articles