Issue
When using the Vault Agent Injector in a Kubernetes environment to generate dynamic credentials from HCP Vault Dedicated, the following error is observed in the Vault Agent logs:
Background
This issue occurs during lease renewal attempts for dynamic credentials generated by HCP Vault. While Vault Agent caching is enabled, the error suggests that the lease is no longer valid—either due to early expiration or unsuccessful renewal.
It’s important to note:
-
Vault Agent Injector only modifies a pod or deployment if specific annotations are present. Refer to the official Vault Injector Annotations documentation for the full list.
Workarounds
1. Use lease_renewal_threshold with a ConfigMap
To better control when lease renewals occur, set the lease_renewal_threshold parameter in the Vault Agent configuration file. This determines how early the agent should attempt renewal before the lease expires.
Note:
lease_renewal_thresholdis not currently available via annotations in Kubernetes.
Steps: Create the Configmap with Vault Agent configuration and Reference the ConfigMap in your deployment using the annotation as
vault.hashicorp.com/agent-configmap: "<config-map-name>"Please find below the sample config map and deployment file used:-
ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-agent-cm
namespace: vault-agent
data:
config.hcl: |
pid_file = "/home/vault/pidfile"
auto_auth {
method "kubernetes" {
mount_path = "auth/kubernetes"
exit_on_err = true
config = {
role = "autoauth"
}
}
sink "file" {
config = {
path = "/home/vault/.vault-token"
}
}
}
template_config {
exit_on_retry_failure = true
static_secret_render_interval = "60m"
max_connections_per_host = 20
lease_renewal_threshold = 0.8
}
template {
contents = "{{- with secret \"database/creds/my-role\" }}postgres://{{ .Data.username }}:{{ .Data.password }}@postgres:5432/postgres?sslmode=disable{{- end }}"
destination = "/home/vault/secrets/db-creds"
error_on_missing_key = true
}
cache {
persist "kubernetes" {
path = "/vault/agent-cache/"
keep_after_import = true
exit_on_err = false
}
}
config-init.hcl: |
exit_after_auth = true
pid_file = "/home/vault/pidfile"
auto_auth {
method "kubernetes" {
mount_path = "auth/kubernetes"
exit_on_err = true
config = {
role = "autoauth"
}
}
sink "file" {
config = {
path = "/home/vault/.vault-token"
}
}
}
template_config {
exit_on_retry_failure = true
static_secret_render_interval = "60m"
max_connections_per_host = 20
lease_renewal_threshold = 0.8
}
template {
contents = "{{- with secret \"database/creds/my-role\" }}postgres://{{ .Data.username }}:{{ .Data.password }}@postgres:5432/postgres?sslmode=disable{{- end }}"
destination = "/home/vault/secrets/db-creds"
error_on_missing_key = true
}
cache {
persist "kubernetes" {
path = "/vault/agent-cache/"
exit_on_err = false
}
}
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: vault-agent
labels:
app: vault-agent-sidecar
spec:
replicas: 1
selector:
matchLabels:
app: vault-agent-sidecar
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "updated"
vault.hashicorp.com/agent-configmap: "vault-agent-cm"
vault.hashicorp.com/agent-image: "hashicorp/vault:latest"
vault.hashicorp.com/agent-cache-enable: "true"
vault.hashicorp.com/log-level: "trace"
labels:
app: vault-agent-sidecar
spec:
containers:
- name: nginx-container
image: nginx:latest
ports:
- containerPort: 80
2. Adjust Default and Maximum TTL for Credentials
Another workaround is to tune the TTL values of the dynamic secrets to minimize lease renewal issues:
-
Set the default TTL to approximately 80% of the maximum TTL.
-
Alternatively, configure the max TTL to exceed the typical application uptime, reducing the likelihood that lease renewals are even needed.
Additional Notes
-
A Pull Request (PR) has already been raised to support the
lease_renewal_thresholdparameter as an injectable annotation in the Vault Agent Injector. Once merged, this should provide a more native solution within the Kubernetes environment.
References
https://developer.hashicorp.com/vault/tutorials/integrate-kubernetes-hcp-vault-dedicated/kubernetes-hcp-vault#vault-agent-injector-service
https://developer.hashicorp.com/vault/docs/platform/k8s/injector/annotations
https://developer.hashicorp.com/vault/tutorials/db-credentials/database-secrets