This article supplements the existing Vault Audit Backends documentation with a description of Vault’s blocked audit logging behavior along with some tips and techniques to recover from this condition.
Understanding Vault Behavior When Audit Logging is Blocked
Vault’s default behavior by design is to no longer service any request if it determines that it cannot write to at least one of the configured Audit Backends.
There is no workaround for this behavior that can be attempted, so it is important that at least one configured backend is writable by Vault before it can resume operations again.
This behavior is documented in the Blocked Audit Backends section of the Audit Backends documentation.
If you are facing this issue, your first goal should be to restore Vault’s ability to write to at least one configured audit log backend.
Blocked File Backend
Vault behaves differently with the file backend than with the socket or syslog backends. If the file handle originally acquired when Vault opened the file is still held, Vault will continue writing to it even when the file is moved to allow for rotation. Vault continues to write when it holds the file handle, so removal of a file audit backend’s file does not cause Vault to cease responding to operations as with the socket or syslog backends.
This is a subtle difference that you should be aware of. Vault does not continuously poll for the existence of the audit log file, and thus cannot know if the file has been removed.
Blocked Socket Backend
Demonstration
To demonstrate this behavior in action, we can start netcat
listening on a TCP port, then configure Vault to use this port as its sole audit logging backend.
Here is a normal Vault startup, along with enabling of the socket audit backend with:
$ vault audit-enable socket address="127.0.0.1:9999" socket_type="tcp"
Successfully enabled audit backend 'socket' with path 'socket'!
==> Vault server configuration:
Cgo: disabled
Cluster Address: https://172.17.0.7:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: true
Redirect Address: http://172.17.0.7:8200
Storage: consul (HA available)
Version: Vault v0.8.3
Version Sha: 6b29fb2b7f70ed538ee2b3c057335d706b6d4e36
==> Vault server started! Log data will stream in below:
...
...
We can confirm the audit log setup by examining our terminal where netcat is listening:
{"time":"2017-11-08T22:13:12Z","type":"response","auth":{"client_token":"hmac-sha256:4ed309f4d13adc231ec06fb5aed92ca5e929d0b24997fd6ba786710d4633ae7b","accessor":"hmac-sha256:80005d798d0a6e4c3e598b2241df687cc0c81059b0895851c62155729ee30219","display_name":"root","policies":["root"],"metadata":null,"entity_id":""},"request":{"id":"0a1f4e72-fa04-4f68-2138-12ec7cc559cd","operation":"update","client_token":"hmac-sha256:4ed309f4d13adc231ec06fb5aed92ca5e929d0b24997fd6ba786710d4633ae7b","client_token_accessor":"hmac-sha256:80005d798d0a6e4c3e598b2241df687cc0c81059b0895851c62155729ee30219","path":"sys/audit/socket","data":{"description":"hmac-sha256:084219485d56777bdea0d3b451e1aaa2af9a82ac983a1178e264f626e0075802","local":false,"options":{"address":"hmac-sha256:b4937a822713f74d97649858f08466835fd263682cc233d908270938793f695f","socket_type":"hmac-sha256:d149336ca20f8de86bf112a4611b47d817d7c73c9bd9805ccdbd79b69103f045"},"type":"hmac-sha256:bdb4a709899d7ed5664a986a667d860aa978bcafecc53ed16e7eb1e1a03ab87e"},"remote_address":"127.0.0.1","wrap_ttl":0,"headers":{}},"response":{},"error":""}
We observe one audit log entry.
Now while Vault is still operating, we’ll stop the netcat
process and observe the logs to learn how Vault reacts after no longer being able to write to this audit log socket:
2017/11/08 22:14:55.993806 [ERROR] core: failed to audit response: request_path=auth/token/create error=1 error occurred:
* no audit backend succeeded in logging the response
Vault is clearly telling us that no audit backend was successful in logging Vault’s response.
If you have a monitoring solution that parses logs, you might consider configuring an alert for occurrences of no audit backend succeeded in logging the response
in that solution so that you’ll have a heads up if this ever occurs.
From the CLI perspective, the error looks like this:
$ vault token-create
Error creating token: Error making API request.
URL: POST http://localhost:8200/v1/auth/token/create
Code: 500. Errors:
* internal error
Vault emits a 500 Internal Server Error code in these cases.
Based on this log and CLI output, your primary task for resolution is to unblock the log backend, and you can expect Vault to return to immediate service having done so. The next section details a technique for unblocking the audit backend in an emergency situation.
Emergency Unblocking of Socket Backend
If you know the configuration (primarily port/protcol) of the socket backed, you can use a utility like netcat
and creative /etc/hosts
file editing to emulate the backend and direct Vault to it.
For example, if the socket backend Options
are example.com:8009
with socket_type
tcp, then you could simply add an entry to the Vault server’s /etc/hosts
file like this:
127.0.0.1 example.com
to direct DNS resolution of the example.com
FQDN to localhost, and then run netcat on the Vault server like this:
$ netcat -l 8009
Vault should be able to connect to the temporary audit log socket provided by the netcat
listener and begin logging to this socket immediately.
You can then authenticate as needed to disable the bad backend config with vault audit-disable
.
If you do not know the details of the audit log backend, you can start Vault version 0.9.0+ with debug level logging, and that configuration will be emitted in the log when Vault is unsealed.
A Note About Audit Log Best Practices
This is a good time to mention that it is always a Vault best practice to enable multiple audit log backends for your Vault servers so that Vault operations do not cease when one of the audit log backends becomes unavailable.
Either avoid relying only one audit log backend type, or if you must use just one type, be sure to use targets which are physically separate (i.e. file based audit logging to two different physical block devices).