This article supplements the existing Vault Audit Backends documentation with a description of Vault’s blocked audit logging behavior along with some tips and techniques to recover from this condition.
Understanding Vault Behavior When Audit Logging is Blocked
Vault’s default behavior by design is to no longer service any request if it determines that it cannot write to at least one of the configured Audit Backends.
There is no workaround for this behavior that can be attempted, so it is important that at least one configured backend is writable by Vault before it can resume operations again.
This behavior is documented in the Blocked Audit Backends section of the Audit Backends documentation.
If you are facing this issue, your first goal should be to restore Vault’s ability to write to at least one configured audit log backend.
Blocked File Backend
Vault behaves differently with the file backend than with the socket or syslog backends. If the file handle originally acquired when Vault opened the file is still held, Vault will continue writing to it even when the file is moved to allow for rotation. Vault continues to write when it holds the file handle, so removal of a file audit backend’s file does not cause Vault to cease responding to operations as with the socket or syslog backends.
This is a subtle difference that you should be aware of. Vault does not continuously poll for the existence of the audit log file, and thus cannot know if the file has been removed.
Blocked Socket Backend
To demonstrate this behavior in action, we can start
netcat listening on a TCP port, then configure Vault to use this port as its sole audit logging backend.
Here is a normal Vault startup, along with enabling of the socket audit backend with:
$ vault audit-enable socket address="127.0.0.1:9999" socket_type="tcp" Successfully enabled audit backend 'socket' with path 'socket'!
==> Vault server configuration: Cgo: disabled Cluster Address: https://172.17.0.7:8201 Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled") Log Level: info Mlock: supported: true, enabled: true Redirect Address: http://172.17.0.7:8200 Storage: consul (HA available) Version: Vault v0.8.3 Version Sha: 6b29fb2b7f70ed538ee2b3c057335d706b6d4e36 ==> Vault server started! Log data will stream in below: ... ...
We can confirm the audit log setup by examining our terminal where netcat is listening:
We observe one audit log entry.
Now while Vault is still operating, we’ll stop the
netcat process and observe the logs to learn how Vault reacts after no longer being able to write to this audit log socket:
2017/11/08 22:14:55.993806 [ERROR] core: failed to audit response: request_path=auth/token/create error=1 error occurred: * no audit backend succeeded in logging the response
Vault is clearly telling us that no audit backend was successful in logging Vault’s response.
If you have a monitoring solution that parses logs, you might consider configuring an alert for occurrences of
no audit backend succeeded in logging the response in that solution so that you’ll have a heads up if this ever occurs.
From the CLI perspective, the error looks like this:
$ vault token-create Error creating token: Error making API request. URL: POST http://localhost:8200/v1/auth/token/create Code: 500. Errors: * internal error
Vault emits a 500 Internal Server Error code in these cases.
Based on this log and CLI output, your primary task for resolution is to unblock the log backend, and you can expect Vault to return to immediate service having done so. The next section details a technique for unblocking the audit backend in an emergency situation.
Emergency Unblocking of Socket Backend
If you know the configuration (primarily port/protcol) of the socket backed, you can use a utility like
netcat and creative
/etc/hosts file editing to emulate the backend and direct Vault to it.
For example, if the socket backend
socket_type tcp, then you could simply add an entry to the Vault server’s
/etc/hosts file like this:
to direct DNS resolution of the
example.com FQDN to localhost, and then run netcat on the Vault server like this:
$ netcat -l 8009
Vault should be able to connect to the temporary audit log socket provided by the
netcat listener and begin logging to this socket immediately.
You can then authenticate as needed to disable the bad backend config with
If you do not know the details of the audit log backend, you can start Vault version 0.9.0+ with debug level logging, and that configuration will be emitted in the log when Vault is unsealed.
A Note About Audit Log Best Practices
This is a good time to mention that it is always a Vault best practice to enable multiple audit log backends for your Vault servers so that Vault operations do not cease when one of the audit log backends becomes unavailable.
Either avoid relying only one audit log backend type, or if you must use just one type, be sure to use targets which are physically separate (i.e. file based audit logging to two different physical block devices).