Introduction
This article covers how to check for and resolve a common configuration error with the Vault Kubernetes auth method which can cause the Vault Agent to hang on startup and return a context deadline exceeded error when it tries to authenticate with Vault.
Problem
When using the Vault Agent Sidecar Injector or a Vault Agent to authenticate against a Kubernetes auth method within Vault, you may see the agent hang and eventually return an error like seen below.
❯ kubectl logs deployment-6d5f56977-l5c2h vault-agent-init -f 04:17:01 PM
2022-02-04T22:16:52.047Z [INFO] sink.file: creating file sink
2022-02-04T22:16:52.047Z [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-02-04T22:16:52.047Z [INFO] auth.handler: starting auth handler
==> Vault agent started! Log data will stream in below:
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.9.0
2022-02-04T22:16:52.047Z [INFO] sink.server: starting sink server
2022-02-04T22:16:52.047Z [INFO] template.server: starting template server
2022-02-04T22:16:52.047Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-02-04T22:16:52.047Z [INFO] auth.handler: authenticating
2022-02-04T22:16:52.048Z [INFO] (runner) creating watcher
2022-02-04T22:17:52.006Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2022-02-04T22:17:53.007Z [INFO] auth.handler: authenticating
2022-02-04T22:18:52.966Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.58s
Cause
If network connectivity between the Kubernetes clients and the Vault server and between Vault server the Kubernetes cluster's API is good and you receive this error, it typically indicates the bound_service_account_names and or bound_service_account_namespaces configured within the auth method's role that the Kubernetes client is using to log in does not contain the appropriate Kubernetes service account name or namespace the Kubernetes client is using.
In the error example above a basic Kubernetes deployment was executed using the default service account name and default namespace, however, the Kubernetes auth method's role it was using was intentionally misconfigured to demonstrate showing the Vault Agent to hang and return an error when it tries to authenticate to Vault.
Below is the Kubernetes deployment configuration. Notice a service account or namespace is not defined, therefore, the default will be used.
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-inject-secret-secretfile: 'kv/secret'
vault.hashicorp.com/role: 'test-role'
vault.hashicorp.com/namespace: "admin"
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
❯ vault read auth/kubernetes/role/test-role
Key Value
--- -----
alias_name_source serviceaccount_uid
bound_service_account_names [accountA]
bound_service_account_namespaces [namespaceB]
policies [policy]
token_bound_cidrs []
token_explicit_max_ttl 0s
token_max_ttl 0s
token_no_default_policy false
token_num_uses 0
token_period 0s
token_policies [policy]
token_ttl 24h
token_type default
ttl 24h
Solution
To resolve you must set the parameters within the role appropriately. You can either explicitly only set the service account name(s) and namespace(s) needed or even use "*" as in some cases it can help as a quick test to validate and you can configure it to be more explicit later as needed. For my example deployment, I am using the default service account and namespace, so I will set it to default.
❯ vault write auth/kubernetes/role/test-role \
bound_service_account_names="default" \
bound_service_account_namespaces="default" \
policies="policy" \
ttl=24h
After re-executing the same Kubernetes deployment from above, the Vault Agent now successfully authenticates and fetches a secret.
❯ kubectl logs deployment-6d5f56977-66xzh vault-agent-init -f 05:03:08 PM
==> Vault agent started! Log data will stream in below:
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.9.0
2022-02-04T23:03:04.864Z [INFO] sink.file: creating file sink
2022-02-04T23:03:04.864Z [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-02-04T23:03:04.864Z [INFO] template.server: starting template server
2022-02-04T23:03:04.864Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-02-04T23:03:04.865Z [INFO] (runner) creating watcher
2022-02-04T23:03:04.865Z [INFO] auth.handler: starting auth handler
2022-02-04T23:03:04.865Z [INFO] auth.handler: authenticating
2022-02-04T23:03:04.865Z [INFO] sink.server: starting sink server
2022-02-04T23:03:05.127Z [INFO] auth.handler: authentication successful, sending token to sinks
2022-02-04T23:03:05.127Z [INFO] auth.handler: starting renewal process
2022-02-04T23:03:05.127Z [INFO] sink.file: token written: path=/home/vault/.vault-token
2022-02-04T23:03:05.127Z [INFO] sink.server: sink server stopped
2022-02-04T23:03:05.127Z [INFO] sinks finished, exiting
2022-02-04T23:03:05.127Z [INFO] template.server: template server received new token
2022-02-04T23:03:05.127Z [INFO] (runner) stopping
2022-02-04T23:03:05.127Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-02-04T23:03:05.127Z [INFO] (runner) creating watcher
2022-02-04T23:03:05.127Z [INFO] (runner) starting
2022-02-04T23:03:05.160Z [INFO] auth.handler: renewed auth token
2022-02-04T23:03:05.229Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/secretfile"
2022-02-04T23:03:05.229Z [INFO] (runner) stopping
2022-02-04T23:03:05.229Z [INFO] template.server: template server stopped
2022-02-04T23:03:05.229Z [INFO] (runner) received finish
2022-02-04T23:03:05.229Z [INFO] auth.handler: shutdown triggered, stopping lifetime watcher
2022-02-04T23:03:05.229Z [INFO] auth.handler: auth handler stopped