Where are My Vault Logs and How do I Share Them with HashiCorp Support?

Avatar
Brian Shumate
  • Updated

Table of Contents

This guide describes the two types of log output a Vault server produces, and provides guidance on locating those log outputs for sharing as part of the support issue troubleshooting workflow.

If you are unsure about your Vault server logs, this guide will help you to identify and share them with HashiCorp Technical Support Engineers when working on Vault issues.

What are the Vault Operational Logs and Where Can I Find Them?

Technical Support Engineers will often ask for Vault operational logs as a troubleshooting step, so it is extremely helpful to us if you provide these logs whenever relevant and especially when opening a new support issue.

If you need to share Vault operational logs and are unsure about their location or format, please read on!

Understanding Vault Operational Logs

Vault is a contemporary server application designed to align well with paradigms such as the Twelve Factor App and contemporary operating system features like journald.

As such, it logs details about its internal operation and subsystem to standard output and standard error.

Here is a sample startup of Vault from the operational logging perspective:

==> Vault server configuration:

             Api Address: http://127.0.0.1:8222
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8223
              Listener 1: tcp (addr: "127.0.0.1:8222", cluster address: "127.0.0.1:8223", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: (not set)
                   Mlock: supported: false, enabled: false
                 Storage: inmem_transactional_ha (HA available)
                 Version: Vault v0.11.3
             Version Sha: fb601237bfbe4bc16ff679f642248ee8a86e627b

==> Vault server started! Log data will stream in below:

2018-10-19T15:26:27.858-0400 [INFO ] core: security barrier not initialized
2018-10-19T15:26:27.858-0400 [INFO ] core: security barrier initialized: shares=1 threshold=1
2018-10-19T15:26:27.879-0400 [INFO ] core: post-unseal setup starting
2018-10-19T15:26:27.887-0400 [INFO ] core: loaded wrapping token key
2018-10-19T15:26:27.887-0400 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018-10-19T15:26:27.887-0400 [INFO ] core: no mounts; adding default mount table
2018-10-19T15:26:27.888-0400 [INFO ] core: successfully mounted backend: type=kv path=secret/

Some points of interest from the above example:

  • The first line of Vault’s operational logging always begins with: ==> Vault server configuration:
    • This line and it subsequent lines are only logged at Vault startup time so they contain vital configuration detail. It’s important to share as much log detail as possible, especially those first 20 lines as part of the complete log output as those contain useful details specific to your Vault instance
  • After configuration lines, Vault emits ==> Vault server started! Log data will stream in below: and switches to a standard log format
  • The standard operational log lines contain time, log level, Vault internal package name/log source, and log message

For a deeper dive into operation details, check out Audit and Operational Log Details.

Finding Operational Logs on Linux Systems

On modern systemd based Linux distributions, the journald daemon will capture Vault’s log output automatically to the system journal and is a common operational logging use case.

You can access a Vault server and issue a quick command to find only the Vault-specific logs entries from the system journal. Presuming your Vault service is named vault, use a command like this to retrieve only those log entries:

$ journalctl -b --no-pager -u vault
...
Oct 15 17:01:47 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:47.950Z [DEBUG] replication.index.local: saved checkpoint: num_dirty=0
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.907Z [DEBUG] rollback: attempting rollback: path=auth/token/
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.907Z [DEBUG] rollback: attempting rollback: path=secret/
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.907Z [DEBUG] rollback: attempting rollback: path=sys/
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.907Z [DEBUG] rollback: attempting rollback: path=identity/
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.907Z [DEBUG] rollback: attempting rollback: path=cubbyhole/
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.947Z [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
Oct 15 17:01:52 ip-10-42-0-27 vault[7954]: 2018-10-15T17:01:52.950Z [DEBUG] replication.index.local: saved checkpoint: num_dirty=0

The output should go back to the system boot time and will sometimes also include restarts of Vault.

If you observe output from the above command that includes log lines prefixed with vault[NNNN]: then you have found the operational logging and can package it up to share in the support ticket.

A convenient command for doing so looks like:

$ journalctl -b --no-pager -u vault | gzip -9 > /tmp/"$(hostname)-$(date +%Y-%m-%dT%H-%M-%SZ)-vault.log.gz"

which result in a compressed log file in the /tmp directory named like this:

/tmp/ip-10-42-0-27-2018-10-15T17:06:49Z-vault.log.gz

If your Vault systemd service is not named “vault” and you are unsure of the service name, then a more generic command can be used:

$ journalctl -b | awk '$5 ~ "vault"'

Please obtain the compressed log file(s) for each Vault server or as directed and share them in the ticket.

NOTE: Vault does not log secrets or related sensitive information, but should you need to sanitize the log content, please let us know that you have done so. Anything shared with HashiCorp Technical Support Engineering through the Support Portal is securely shared via SendSafely and kept encrypted at rest.

Static File Logging

If you do not observe vault[NNNN]: style lines in your output from the previous command example, your Vault startup script could be instead sending the log output elsewhere. To learn more, take a look into the Vault systemd unit, which is often (but not always) located at /etc/systemd/system/vault.service.

If you examine the unit file and notice something like:

...
[Service]
...
ExecStart=/bin/sh -c '/home/vagrant/bin/vault server -config=/home/vagrant/vault_nano/config/vault -log-level="trace" > /var/log/vault.log
...

You can safely presume that Vault is actually storing its operational logging in the static file located at /var/log/vault.log, which you can confirm with sufficient privileges:

$ sudo tail -f /var/log/vault.log
2018-10-19T19:04:16.179Z [TRACE] pkcs11: pkcs11 mechanism selected: mechanism=0x1085 name=aes-cbc-pad
2018-10-19T19:04:16.179Z [TRACE] pkcs11: pkcs11 hmac mechanism selected: mechanism=0x251 name=sha256
2018-10-19T19:04:16.180Z [TRACE] pkcs11: key successfully found
2018-10-19T19:04:16.180Z [TRACE] pkcs11: HMAC key successfully found
2018-10-19T19:04:16.182Z [TRACE] core: activating sealwrap capability
2018-10-19T19:04:16.182Z [TRACE] core: initializing licensing
2018-10-19T19:04:16.183Z [DEBUG] storage.cache: creating LRU cache: size=0
2018-10-19T19:04:16.213Z [DEBUG] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
2018-10-19T19:04:16.214Z [INFO ] core: stored unseal keys supported, attempting fetch
2018-10-19T19:04:16.215Z [WARN ] core: stored unseal key(s) supported but none found

If you have recent valid log entries, please use a command like the following to share them with us:

$ sudo tar zcvf  /tmp/"$(hostname)-$(date +%Y-%m-%dT%H:%M:%SZ)-vault-logs.tgz" /var/log/vault.log*
tar: Removing leading `/' from member names
/var/log/vault.log
/var/log/vault.log.1
/var/log/vault.log.2
/var/log/vault.log.3

The file in /tmp will have a filename like this:

/tmp/vault-nano-dc1-2018-10-19T19:18:29Z-vault-logs.tgz

Please share the file output with us directly in the support ticket.

Other System Log Locations

If Vault is not operating on on Linux or is not operating on a systemd based Linux, it could be configured to log to the system log via a facility like logger, and so Vault’s logs could be part of the main system logs in these locations:

  • /var/log/messages
  • /var/log/syslog

You can execute a quick grep command to check those locations:

$ sudo grep 'vault\[' /var/log/messages || \
  sudo grep 'vault\[' /var/log/syslog

Vault could also (though rarely) be configured to log to a static file via other init systems, such as SystemV-style init or Upstart. For example, in a SystemV-style init script like /etc/init.d/vault might contain clues as to the log file location.

Docker

Logs from Vault Docker containers can be retrieved with the docker logs command as in this example:

$ docker logs vault0
Using eth0 for VAULT_REDIRECT_ADDR: http://172.17.0.8:8200
Using eth0 for VAULT_CLUSTER_ADDR: https://172.17.0.8:8201
==> Vault server configuration:

             Api Address: http://172.17.0.8:8200
                     Cgo: disabled
         Cluster Address: https://172.17.0.8:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
                 Storage: consul (HA available)
                 Version: Vault v1.0.3+prem
             Version Sha: 28a8a6dae0cea127d92e6f2a3cb0153132ab8c13

==> Vault server started! Log data will stream in below:

2019-03-12T20:15:44.680Z [DEBUG] storage.consul: config path set: path=vault/
2019-03-12T20:15:44.680Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2019-03-12T20:15:44.680Z [DEBUG] storage.consul: config service set: service=vault
2019-03-12T20:15:44.680Z [DEBUG] storage.consul: config service_tags set: service_tags=secrets
...

Kubernetes

Logs from Vault Kubernetes pods can be retrieved with the kubectl logs command as in this example:

$ kubectl logs vault-55bcb779b4-8mfn6
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config path set: path=vault
2019-03-13T13:54:20.702Z [WARN]  storage.consul: appending trailing forward slash to path
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config service set: service=vault
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config service_tags set: service_tags=
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config service_address set: service_address=<nil>
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config check_timeout set: check_timeout=30s
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config address set: address=127.0.0.1:8443
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: config scheme set: scheme=https
2019-03-13T13:54:20.702Z [DEBUG] storage.consul: configured TLS
...

What are the Vault Audit Device Logs and Where Can I Find Them?

Technical Support Engineers will sometimes ask for Vault audit device logs as a troubleshooting step; while these logs are not as commonly requested as operational logging, it helps to know how to locate and share them when needed.

If you need to share Vault audit device logs and are unsure about their location or format, please read on!

Understanding Vault Audit Device Logs

The Vault Audit Device represent raw request and response entries which HMAC sensitive fields. The audit device logs are configured within Vault itself, so basic information about their configuration can be inspected with the API, CLI, or web UI.

In their raw form, they’re JSON data, described in more detail in Audit and Operational Log Details.

Listing Audit Devices and Locating Audit Log Files

Use the -detailed flag to get the full path to get audit device options which will in the case of the file device, include the full path to the audit device log file:

$ vault audit list -detailed
Path     Type    Description                         Replication    Options
----     ----    -----------                         -----------    -------
file/    file    An example file audit device        replicated     file_path=/var/log/vault/vault_audit.log

Here we can observe that /var/log/vault/vault_audit.log would be the audit device log file of interest.

Sharing a compressed instance of such logging when audit logs are requested or when the issue deals specifically with audit devices and logs is always helpful.

If the audit device is socket or syslog based, you’ll have to retrieve the audit logs from the system that is collecting them via socket, or retrieve them from the system log accordingly.

If the audit logs are being sent to a local syslog, they should be available for example in /var/log/messages or /var/log/syslog.

Audit Log Entries

A typical entry is a compressed JSON object, one for each request and response, that looks like this:

{"time":"2018-10-30T17:09:16.5828352Z","type":"response","auth":{"client_token":"hmac-sha256:b64a134ba05714a4644e27502e1d0f10f162b2b49bc16772053759511c448c14","accessor":"hmac-sha256:05a813e1e80ca67bfae7f61cca2cbaa36c6060a6a03ff962ed80a61b66c1cabf","display_name":"root","policies":["root"],"token_policies":["root"],"metadata":null,"entity_id":""},"request":{"id":"4a08e537-8f23-1861-6962-15c0492363d9","operation":"update","client_token":"hmac-sha256:b64a134ba05714a4644e27502e1d0f10f162b2b49bc16772053759511c448c14","client_token_accessor":"hmac-sha256:05a813e1e80ca67bfae7f61cca2cbaa36c6060a6a03ff962ed80a61b66c1cabf","namespace":{"id":"root","path":""},"path":"sys/audit/file","data":{"description":"hmac-sha256:2c8f6620b5e06d75690edd867848412fc422fc2ad35cdcb55bff9216d06717c5","local":false,"options":{"file_path":"hmac-sha256:467918fdeaccf2c2e03417804dfcbb5ae9eec6f387a97337fb68e151ce886e00"},"type":"hmac-sha256:fb1ee48d703b363d62d36d17b3e59c647193295a029230805399485bb9ed9e21"},"policy_override":false,"remote_address":"172.17.0.1","wrap_ttl":0,"headers":{}},"response":{},"error":""}

A quick way to find these entries is to grep for client_token:

$ sudo grep client_token /var/log/messages \
  || sudo grep client_token /var/log/syslog

if it is a systemd based distribution, then the audit logs captured by syslog will likely be in the journal, which you can confirm with something like:

$ journalctl | awk '$5 ~ "vault"' | grep client_token

As with the operational logs, please compress and share audit logs when requested.

If you want to take a deeper dive into Vault logging, be sure to check out the articles listed in Resources.

Thanks!

Resources

  1. Audit and Operational Log Details
  2. Audit Log Notes
  3. Audit Devices
  4. File Audit Device
  5. Syslog Audit Device
  6. Socket Audit Device

Articles in this section

See more

Related articles