Vault Agent Caching allows client-side caching of responses containing newly created tokens and responses containing leased secrets generated off of these newly created tokens. The renewals of the cached tokens and leases are also managed by the agent.
Vault Agent can restore tokens and leases from a persistent cache file created by a previous Vault Agent process. The persistent cache is a BoltDB file that includes tuples encrypted by a generated encryption key. The encrypted tuples include the Vault token used to retrieve secrets, leases for tokens/secrets, and secret values.
As of writing this article, the Vault Agent persistent cache is currently
supported only in a Kubernetes environment.
This article demonstrates how Vault agent "Persistent Cache" works, and how it handover credentials generated from one agent process to another.
Prerequisite:
- A running Vault cluster (Any stable release) alongside an Agent Injector Webhook pod on a Kubernetes cluster.
- You can also use Vault hosted external to the Kubernetes cluster by setting externalvaultaddr.
- Properly configured a Kubernetes auth method & a dynamic credential provider secret engine to be used by the Vault agent that will run alongside the application container.
- In this article, we use PostgreSQL as a dynamic credential provider.
- Please note that the Vault init and agent customized configurations are passed from the annotations only, they can also passed using ConfigMaps via the annotation
"vault.hashicorp.com/agent-configmap".
Cache in action:
Please refer to the following block of yaml code (bare minimum but not exhaustive) to deploy a sample NGINX application with cache enabled via an annotation that will trigger the webhook to deploy an init container first that will generate the auth token & dynamic credentials which will then be handed over to the agent container with the help of persistent cache.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: default
labels:
app: vault-agent-sidecar
spec:
replicas: 1
selector:
matchLabels:
app: vault-agent-sidecar
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "updated"
vault.hashicorp.com/agent-image: "hashicorp/vault:latest"
vault.hashicorp.com/agent-cache-enable: "true"
vault.hashicorp.com/log-level: "trace"
vault.hashicorp.com/agent-inject-template-vault.yaml: |
{{- with secret "database/creds/my-role" }}
{{ printf "db_username:" }} {{ .Data.username }}
{{ printf "db_password:" }} {{ .Data.password }}
{{- end }}
vault.hashicorp.com/role: autoauth
labels:
app: vault-agent-sidecar
spec:
containers:
- name: nginx-container
image: nginx:latest
ports:
- containerPort: 80
-> Here is how the deployment pods will look like post the "kubectl apply" of the above yaml code:
% k get po -n default
NAME READY STATUS RESTARTS AGE
nginx-deployment-5fc7889f79-74nd5 2/2 Running 0 11m
vault-agent-injector-6db6d778cd-6cxkh 1/1 Running 0 17h
-> Here is what the init container logs will look like:
~ % k logs -f nginx-deployment-5fc7889f79-74nd5 -n default -c vault-agent-init
===> Vault Agent started! Log data will stream in below:
2025-02-21T08:22:21.527Z [INFO] agent.sink.file: creating file sink
2025-02-21T08:22:21.527Z [TRACE] agent.sink.file: enter write_token: path=/home/vault/.vault-token
2025-02-21T08:22:21.527Z [TRACE] agent.sink.file: exit write_token: path=/home/vault/.vault-token
2025-02-21T08:22:21.527Z [INFO] agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r----- owner=100 group=1000
2025-02-21T08:22:21.528Z [INFO] agent.cache: configured persistent storage: path=/vault/agent-cache/
2025-02-21T08:22:21.528Z [DEBUG] agent.apiproxy: auto-auth token is allowed to be used; configuring inmem sink
==> Vault Agent configuration:
Api Address 1: http://bufconn
Cgo: disabled
Log Level: trace
Version: Vault v1.18.4, built 2025-01-29T13:57:54Z
Version Sha: 503be623a3697e8c10c254dc87d26492c46753e1
2025-02-21T08:22:21.528Z [DEBUG] agent: would have sent systemd notification (systemd not present): notification=READY=1
2025-02-21T08:22:21.528Z [INFO] agent.exec.server: starting exec server
2025-02-21T08:22:21.528Z [INFO] agent.exec.server: no env templates or exec config, exiting
2025-02-21T08:22:21.528Z [INFO] agent.auth.handler: starting auth handler
2025-02-21T08:22:21.528Z [INFO] agent.auth.handler: authenticating
2025-02-21T08:22:21.528Z [TRACE] agent.auth.kubernetes: beginning authentication
2025-02-21T08:22:21.528Z [INFO] agent.sink.server: starting sink server
2025-02-21T08:22:21.528Z [INFO] agent.template.server: starting template server
2025-02-21T08:22:21.528Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
2025-02-21T08:22:21.529Z [DEBUG] agent: (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"TRACE","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"{{- with secret \"database/creds/my-role\" }}postgres://{{ .Data.username }}:{{ .Data.password }}@postgres:5432/postgres?sslmode=disable{{- end }}","CreateDestDirs":true,"Destination":"/home/vault/secrets/db-creds","ErrMissingKey":true,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":"","MapToEnvironmentVariable":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":{},"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":20,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"ClientUserAgent":null,"DefaultLeaseDuration":3600000000000,"LeaseRenewalThreshold":0.8,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000,"ErrOnFailedLookup":false}
2025-02-21T08:22:21.529Z [INFO] agent: (runner) creating watcher
2025-02-21T08:22:21.563Z [INFO] agent.auth.handler: authentication successful, sending token to sinks
2025-02-21T08:22:21.564Z [TRACE] agent.sink.file: enter write_token: path=/home/vault/.vault-token
2025-02-21T08:22:21.564Z [INFO] agent.sink.file: token written: path=/home/vault/.vault-token
2025-02-21T08:22:21.564Z [TRACE] agent.sink.file: exit write_token: path=/home/vault/.vault-token
2025-02-21T08:22:21.564Z [DEBUG] agent.cache.leasecache: storing auto-auth token into the cache
2025-02-21T08:22:21.564Z [INFO] agent.auth.handler: starting renewal process
2025-02-21T08:22:21.564Z [INFO] agent.template.server: template server received new token
2025-02-21T08:22:21.564Z [INFO] agent: (runner) stopping
2025-02-21T08:22:21.564Z [DEBUG] agent: (runner) stopping watcher
2025-02-21T08:22:21.564Z [DEBUG] agent: (watcher) stopping all views
2025-02-21T08:22:21.564Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
2025-02-21T08:22:21.564Z [TRACE] agent.cache.leasecache: set entry in persistent storage: type=token path=EBNBz id=yBtrL
2025-02-21T08:22:21.564Z [INFO] agent.sink.server: sink server stopped
2025-02-21T08:22:21.565Z [INFO] agent: sinks finished, exiting
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"TRACE","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"{{- with secret \"database/creds/my-role\" }}postgres://{{ .Data.username }}:{{ .Data.password }}@postgres:5432/postgres?sslmode=disable{{- end }}","CreateDestDirs":true,"Destination":"/home/vault/secrets/db-creds","ErrMissingKey":true,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":"","MapToEnvironmentVariable":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":{},"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":20,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"ClientUserAgent":"Vault Agent Templating/1.18.4 (+https://www.vaultproject.io/; go1.23.3)","DefaultLeaseDuration":3600000000000,"LeaseRenewalThreshold":0.8,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000,"ErrOnFailedLookup":false}
2025-02-21T08:22:21.565Z [INFO] agent: (runner) creating watcher
2025-02-21T08:22:21.565Z [INFO] agent: (runner) starting
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) running initial templates
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) initiating run
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) checking template b1477c403ea3bddd9932719eaac328b2
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) missing data for 1 dependencies
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) missing dependency: vault.read(database/creds/my-role)
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) add used dependency vault.read(database/creds/my-role) to missing since isLeader but do not have a watcher
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) was not watching 1 dependencies
2025-02-21T08:22:21.565Z [DEBUG] agent: (watcher) adding vault.read(database/creds/my-role)
2025-02-21T08:22:21.565Z [TRACE] agent: (watcher) vault.read(database/creds/my-role) starting
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) diffing and updating dependencies
2025-02-21T08:22:21.565Z [DEBUG] agent: (runner) watching 1 dependencies
2025-02-21T08:22:21.565Z [TRACE] agent: (view) vault.read(database/creds/my-role) starting fetch
2025-02-21T08:22:21.565Z [INFO] agent.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/database/creds/my-role
2025-02-21T08:22:21.565Z [TRACE] agent.cache.leasecache: checking cache for dynamic secret request: id=d986a089f917033f1fe40b263c5063af05736bf3ae7d53a9e970b13ce0cb628a
2025-02-21T08:22:21.565Z [TRACE] agent.cache.leasecache: checking cache for static secret request: id=10b40258ca8538ae3bc592f1c43876aa39bc2c85e4e972f781a2fe27330b62e6
2025-02-21T08:22:21.565Z [DEBUG] agent.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/database/creds/my-role
2025-02-21T08:22:21.566Z [INFO] agent.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/database/creds/my-role
2025-02-21T08:22:21.566Z [DEBUG] agent.apiproxy.client: performing request: method=GET url=http://x.x.x.x:8200/v1/sys/internal/ui/mounts/database/creds/my-role
2025-02-21T08:22:21.572Z [INFO] agent.auth.handler: renewed auth token
2025-02-21T08:22:21.573Z [DEBUG] agent.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/database/creds/my-role
2025-02-21T08:22:21.573Z [TRACE] agent: vault.read(database/creds/my-role): GET /v1/database/creds/my-role
2025-02-21T08:22:21.574Z [INFO] agent.apiproxy: received request: method=GET path=/v1/database/creds/my-role
2025-02-21T08:22:21.574Z [TRACE] agent.cache.leasecache: checking cache for dynamic secret request: id=49699a4360562061d9dea6366b09d55a2b29df13459ac734c96daba3863cce0d
2025-02-21T08:22:21.574Z [DEBUG] agent.cache.leasecache: forwarding request from cache: method=GET path=/v1/database/creds/my-role
2025-02-21T08:22:21.574Z [INFO] agent.apiproxy: forwarding request to Vault: method=GET path=/v1/database/creds/my-role
2025-02-21T08:22:21.574Z [DEBUG] agent.apiproxy.client: performing request: method=GET url=http://x.x.x.x:8200/v1/database/creds/my-role
2025-02-21T08:22:21.597Z [DEBUG] agent.cache.leasecache: processing lease response: method=GET path=/v1/database/creds/my-role
2025-02-21T08:22:21.597Z [DEBUG] agent.cache.leasecache: storing dynamic secret response into the cache: method=GET path=/v1/database/creds/my-role id=49699a4360562061d9dea6366b09d55a2b29df13459ac734c96daba3863cce0d
2025-02-21T08:22:21.597Z [TRACE] agent.cache.leasecache: set entry in persistent storage: type=lease path=/v1/database/creds/my-role id=49699a4360562061d9dea6366b09d55a2b29df13459ac734c96daba3863cce0d
2025-02-21T08:22:21.598Z [TRACE] agent: (view) vault.read(database/creds/my-role) marking successful data response
2025-02-21T08:22:21.598Z [TRACE] agent: (view) vault.read(database/creds/my-role) successful contact, resetting retries
2025-02-21T08:22:21.598Z [TRACE] agent: (view) vault.read(database/creds/my-role) received data
2025-02-21T08:22:21.598Z [TRACE] agent: (view) vault.read(database/creds/my-role) starting fetch
2025-02-21T08:22:21.598Z [TRACE] agent: vault.read(database/creds/my-role): starting renewer
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) receiving dependency vault.read(database/creds/my-role)
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) initiating run
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) checking template b1477c403ea3bddd9932719eaac328b2
2025-02-21T08:22:21.598Z [DEBUG] agent.cache.leasecache: initiating renewal: method=GET path=/v1/database/creds/my-role
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) rendering "(dynamic)" => "/home/vault/secrets/db-creds"
2025-02-21T08:22:21.598Z [INFO] agent: (runner) rendered "(dynamic)" => "/home/vault/secrets/db-creds"
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) diffing and updating dependencies
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) vault.read(database/creds/my-role) is still needed
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) watching 1 dependencies
2025-02-21T08:22:21.598Z [DEBUG] agent: (runner) all templates rendered
2025-02-21T08:22:21.599Z [INFO] agent: (runner) stopping
2025-02-21T08:22:21.599Z [DEBUG] agent: (runner) stopping watcher
2025-02-21T08:22:21.599Z [DEBUG] agent: (watcher) stopping all views
2025-02-21T08:22:21.599Z [TRACE] agent: (watcher) stopping vault.read(database/creds/my-role)
2025-02-21T08:22:21.599Z [INFO] agent.template.server: template server stopped
2025-02-21T08:22:21.599Z [TRACE] agent: (view) vault.read(database/creds/my-role) stopping poll (received on view stopCh)
2025-02-21T08:22:21.599Z [INFO] agent: (runner) received finish
2025-02-21T08:22:21.598Z [INFO] agent.apiproxy: received request: method=PUT path=/v1/sys/leases/renew
2025-02-21T08:22:21.599Z [TRACE] agent.cache.leasecache: checking cache for dynamic secret request: id=970e782ef9352878933ed101f1303c7d4b6fd1066b9e3c5616a05cb01db5b9d2
2025-02-21T08:22:21.599Z [DEBUG] agent.cache.leasecache: forwarding request from cache: method=PUT path=/v1/sys/leases/renew
2025-02-21T08:22:21.599Z [INFO] agent.apiproxy: forwarding request to Vault: method=PUT path=/v1/sys/leases/renew
2025-02-21T08:22:21.599Z [DEBUG] agent.apiproxy.client: performing request: method=PUT url=http://x.x.x.x:8200/v1/sys/leases/renew
2025-02-21T08:22:21.599Z [ERROR] agent.apiproxy.client: request failed: error="Put \"http://x.x.x.x:8200/v1/sys/leases/renew\": context canceled" method=PUT url=http://x.x.x.x:8200/v1/sys/leases/renew
2025-02-21T08:22:21.599Z [INFO] agent.auth.handler: shutdown triggered, stopping lifetime watcher
2025-02-21T08:22:21.599Z [INFO] agent.auth.handler: auth handler stopped
2025-02-21T08:22:21.599Z [INFO] agent.exec.server: exec server stopped
2025-02-21T08:22:21.599Z [DEBUG] agent: would have sent systemd notification (systemd not present): notification=STOPPING=1
2025-02-21T08:22:21.599Z [TRACE] agent.cache.cacheboltdb: closing bolt db: path=/vault/agent-cache/vault-agent-cache.db
-> Here is what the agent container logs (the very initial part of its startup sequence) will look like:
~ % k logs -f nginx-deployment-5fc7889f79-74nd5 -n default -c vault-agent
==> Vault Agent started! Log data will stream in below:
==> Vault Agent configuration:
Api Address 1: http://bufconn
Cgo: disabled
Log Level: trace
Version: Vault v1.18.4, built 2025-01-29T13:57:54Z
Version Sha: 503be623a3697e8c10c254dc87d26492c46753e1
2025-02-21T08:22:22.352Z [INFO] agent.sink.file: creating file sink
2025-02-21T08:22:22.352Z [TRACE] agent.sink.file: enter write_token: path=/home/vault/.vault-token
2025-02-21T08:22:22.352Z [TRACE] agent.sink.file: exit write_token: path=/home/vault/.vault-token
2025-02-21T08:22:22.352Z [INFO] agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r----- owner=100 group=1000
2025-02-21T08:22:22.353Z [TRACE] agent.cache.cacheboltdb: closing bolt db: path=/vault/agent-cache/vault-agent-cache.db
2025-02-21T08:22:22.353Z [TRACE] agent.cache.leasecache: restored token: id=yBtrL
2025-02-21T08:22:22.353Z [TRACE] agent.cache.leasecache: restoring lease: id=49699a4360562061d9dea6366b09d55a2b29df13459ac734c96daba3863cce0d path=/v1/database/creds/my-role
2025-02-21T08:22:22.353Z [TRACE] agent.cache.leasecache: restored lease: id=49699a4360562061d9dea6366b09d55a2b29df13459ac734c96daba3863cce0d path=/v1/database/creds/my-role
2025-02-21T08:22:22.353Z [INFO] agent.cache: loaded memcache from persistent storage
2025-02-21T08:22:22.353Z [DEBUG] agent.apiproxy: auto-auth token is allowed to be used; configuring inmem sink
2025-02-21T08:22:22.353Z [DEBUG] agent.cache.leasecache: initiating renewal: method=GET path=/v1/database/creds/my-role
2025-02-21T08:22:22.354Z [DEBUG] agent: would have sent systemd notification (systemd not present): notification=READY=1
If you would only follow the trail of "agent.cache" logger, you would see that the auth token and the dynamic credentials generated by the init process were saved in the persistent cache file names "vault-agent-cache.db" at the default path "/vault/agent-cache/" and later they were restored as is by the agent container.
The cache stores the data with a reference ID in the cache file (BoltDB file) and the same ID can be seen restored.