Vault Agent templating behaviour while using WriteTo helper function

Avatar
Yash Verma
  • Updated

Introduction:

Vault Agent is commonly used with the template stanza to render dynamic secrets or certificates to files on disk. The WriteTo functionality Writes the content to a file with permissions, username (or UID), group name (or GID), and optional flags to select appending mode or add a newline. This is particularly helpful when dealing with short-lived PKI certificates or frequently rotating secrets.

However, in certain scenarios where the rendered secret has a longer validity period (TTL) and does not change, the file defined as per WriteTo helper function will still be updated during every auto-auth token renewal event. This behaviour may confuse users since no actual change has occurred in the secret or template content itself.

Summary:

When running Vault Agent with auto-auth enabled and a PKI cert template defined using the WriteTo function, the agent writes to the path at every token renewal interval, even though:

  • The PKI certificate's TTL is still valid.
  • The certificate has not been rotated or reissued.
  • The content in the destination file as per template has not changed.

As a result, the same output file is repeatedly written, which may lead to unnecessary downstream reloads or redundant updates in applications consuming that file.

Below is the source template -->

cat yash.tpl 
{{ with pkiCert "pki_int/issue/example-dot-com" "common_name=example.com" }}
{{ .Data.Key }}{{ .Data.Cert }}{{ .Data.CA }}
{ .Data.Key | writeToFile "/home/ubuntu/private.key" "ubuntu" "ubuntu" "0600" }}
{{ .Data.Cert | writeToFile "/home/ubuntu/public.crt" "ubuntu" "ubuntu" "0644" }}
{{ .Data.CA |  writeToFile "/home/ubuntu/minio.cacrt" "ubuntu" "ubuntu" "0644" }}
{{ end }}

In this configuration:

  • The pkiCert function fetches a certificate from Vault.
  • The WriteTo function writes the Key, Cert, and CA to their respective paths.
  • On each token renewal event, Vault Agent evaluates the template and executes the WriteTo action.

 

Observed Logs -->

2025-08-17T11:52:49.122Z [INFO]  agent.auth.handler: lifetime watcher done channel triggered, re-authenticating
2025-08-17T11:52:50.122Z [INFO]  agent.auth.handler: authenticating
2025-08-17T11:52:50.128Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2025-08-17T11:52:50.128Z [DEBUG] agent.cache.leasecache: storing auto-auth token into the cache
2025-08-17T11:52:50.128Z [INFO]  agent.template.server: template server received new token
2025-08-17T11:52:50.128Z [INFO]  agent: (runner) stopping
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) stopping watcher
2025-08-17T11:52:50.128Z [DEBUG] agent: (watcher) stopping all views
2025-08-17T11:52:50.128Z [TRACE] agent: (watcher) stopping vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem)
2025-08-17T11:52:50.128Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2025-08-17T11:52:50.128Z [INFO]  agent.auth.handler: starting renewal process
2025-08-17T11:52:50.128Z [TRACE] agent: (view) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) stopping poll (received on view stopCh)
2025-08-17T11:52:50.128Z [INFO]  agent: (runner) received finish
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"TRACE","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"/home/ubuntu/yash.pem","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"/home/ubuntu/yash.tpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":"","MapToEnvironmentVariable":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":{},"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":10,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"ClientUserAgent":"Vault Agent Templating/1.19.0+ent (+https://www.vaultproject.io/; go1.23.6)","DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000,"ErrOnFailedLookup":false}
2025-08-17T11:52:50.128Z [INFO]  agent: (runner) creating watcher
2025-08-17T11:52:50.128Z [TRACE] agent.cache.leasecache: auto-auth token already exists in cache; no need to store it again
2025-08-17T11:52:50.128Z [INFO]  agent: (runner) starting
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) running initial templates
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) initiating run
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) checking template 38d93bce158d67d65af475e7415c7ca2
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) missing data for 1 dependencies
2025-08-17T11:52:50.128Z [DEBUG] agent: (runner) missing dependency: vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem)
2025-08-17T11:52:50.129Z [DEBUG] agent: (runner) add used dependency vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) to missing since isLeader but do not have a watcher
2025-08-17T11:52:50.129Z [DEBUG] agent: (runner) was not watching 1 dependencies
2025-08-17T11:52:50.129Z [DEBUG] agent: (watcher) adding vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem)
2025-08-17T11:52:50.129Z [TRACE] agent: (watcher) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) starting
2025-08-17T11:52:50.129Z [DEBUG] agent: (runner) diffing and updating dependencies
2025-08-17T11:52:50.129Z [DEBUG] agent: (runner) watching 1 dependencies
2025-08-17T11:52:50.129Z [TRACE] agent: (view) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) starting fetch
2025-08-17T11:52:50.130Z [INFO]  agent.apiproxy: received request: method=PUT path=/v1/pki_int/issue/example-dot-com
2025-08-17T11:52:50.130Z [TRACE] agent.cache.leasecache: checking cache for dynamic secret request: id=bb3063f1df2736088a43d880447e1f6b9391b789c47d869b2c28471b28216623
2025-08-17T11:52:50.130Z [DEBUG] agent.cache.leasecache: forwarding request from cache: method=PUT path=/v1/pki_int/issue/example-dot-com
2025-08-17T11:52:50.130Z [INFO]  agent.apiproxy: forwarding request to Vault: method=PUT path=/v1/pki_int/issue/example-dot-com
2025-08-17T11:52:50.130Z [DEBUG] agent.apiproxy.client: performing request: method=PUT url=http://172.31.35.102:8200/v1/pki_int/issue/example-dot-com
2025-08-17T11:52:50.459Z [DEBUG] agent.cache.leasecache: pass-through response; secret not renewable: method=PUT path=/v1/pki_int/issue/example-dot-com
2025-08-17T11:52:50.460Z [TRACE] agent: (view) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) marking successful data response
2025-08-17T11:52:50.460Z [TRACE] agent: (view) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) successful contact, resetting retries
2025-08-17T11:52:50.460Z [TRACE] agent: (view) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) received data
2025-08-17T11:52:50.460Z [TRACE] agent: (view) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) starting fetch
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) receiving dependency vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem)
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) initiating run
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) checking template 38d93bce158d67d65af475e7415c7ca2
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) rendering "/home/ubuntu/yash.tpl" => "/home/ubuntu/yash.pem"
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) diffing and updating dependencies
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) vault.pki(pki_int/issue/example-dot-com->/home/ubuntu/yash.pem) is still needed
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) watching 1 dependencies
2025-08-17T11:52:50.460Z [DEBUG] agent: (runner) all templates rendered

 

At every re-authentication to the vault server from the vault-agent using auto-auth function it checks the template destination and writes the respective block of the certificate in the path defined using the WriteTo function. The catch is that it only overwrites the same information in the existing certificate inside the path, which is /home/ubuntu.

 The last modification time of the WriteTo file changes at every token renewal. 

ubuntu@ip-172-31-35-102:~$ pwd
/home/ubuntu
ubuntu@ip-172-31-35-102:~$ ls -lrth
total 80K
-rw-r--r-- 1 root   root    328 Sep  3  2024 config.hcl
drwxr-xr-x 3 ubuntu ubuntu 4.0K Oct 17  2024 vault
-rw-rw-r-- 1 root   root   9.3K Feb 24 17:53 TermsOfEvaluation.txt
-rw-rw-r-- 1 root   root    31K Feb 24 17:53 EULA.txt
-rw-rw-r-- 1 ubuntu ubuntu  665 Aug 12 05:35 root_token.txt
-rw-rw-r-- 1 ubuntu ubuntu 1.2K Aug 16 05:47 root_2023_ca.crt
-rw-rw-r-- 1 ubuntu ubuntu  924 Aug 16 05:48 pki_intermediate.csr
-rw-rw-r-- 1 ubuntu ubuntu 2.5K Aug 16 05:48 intermediate.cert.pem
-rw-rw-r-- 1 ubuntu ubuntu   87 Aug 17 11:29 yash.tpl
-rw-rw-r-- 1 ubuntu ubuntu    2 Aug 17 11:31 yash.pem
-rw------- 1 ubuntu ubuntu 1.7k Aug 17 11:50 private.key
-rw-r--r-- 1 ubuntu ubuntu 1.2k Aug 17 11:50 public.crt
-rw-r--r-- 1 ubuntu ubuntu 1.2k Aug 17 11:50 minio.cacrt
drwxrwxr-x 2 ubuntu ubuntu 4.0K Aug 17 13:32 vault-agent

References:

PKI Cert template

Auto Auth Vault Agent

writeToFile helper function

 

 

Articles in this section

See more

Related articles