Introduction
Problem
There are blocking conditions that may be met when using a SINGLE Syslog Audit Device as a destination and when those are encountering issues; this is also highlighted in the related Audit documentation: Blocked Audit Devices and Audit Device Notes.
Here are some example messages that may be observed in the Vault operational logs as a result of issues:
…… [ERROR] audit: backend failed to log response: backend=syslog/ error="Unix syslog delivery error"
… [ERROR] core: failed to audit response: request_path=sys/audit/syslog
error=
| 1 error occurred:
| \t* no audit backend succeeded in logging the response
|
……… [ERROR] audit: backend failed to log response: backend=syslogtest/ error="Unix syslog delivery error"
……… [ERROR] core: failed to audit response: request_path=sys/audit/syslog
error=
| 1 error occurred:
| \t* no audit backend succeeded in logging the response
|
……… [ERROR] audit: backend failed to log response: backend=<syslog_mount>/ error="Unix syslog delivery error"
……… [ERROR] core: failed to audit response: request_path=sys/audit/syslog
error=
| 1 error occurred:
| \t* no audit backend succeeded in logging the response
|
{"@level":"error","@message":"failed to audit response","@module":"core","@timestamp":"2024-05-02T14:25:47.094393Z","error":"2 errors occurred:\n\t* event not processed by enough 'sink' nodes\n\t* event.(SyslogSink).Process: error writing to syslog: write unixgram @-\u003e/dev/log: write: message too long\n\n","request_path":"sys/internal/counters/activity"}
(*) where <syslog_mount> is the path used for audit.
Prerequisites
- Vault Enterprise (All versions) with single audit destination as local syslog (socket).
Cause
- Failure to write to the local syslog socket is blocking Vault for accepting service requests or configuration commands.
Overview of possible solutions
Solutions
Solution 1: Fix the Audit Device & services will resume on the host in question
The process of Vault is dependent on the availability of syslog socket and will be able to recover ONLY after restoring the syslog daemon service.
Notes:
- Trying to disable the syslog audit destination will NOT be accepted and will throw an error like:
vault audit disable syslog
Error disabling audit device: Error making API request.
URL: DELETE http://127.0.0.1:8200/v1/sys/audit/syslog
Code: 500. Errors:
* internal error
- Please note that it is NOT possible to change the path of the audit destination to a different one even if the new audit destination is a file type and has enough free space to accommodate the recording or requests.
- A possible emergency scenario can include starting a dummy socket listening process using Netcat utility like in the below example. Temporarily enabling this syslog daemon must align with security policies and rules. Please get in contact with your OS Administrator and Security Team before enabling the socket listening.
# As root user, run the following command to enable a dummy socket listening daemon.
# The standard listening port for SYSLOG is 514.
# You may adapt the port as per your configuration
nc -l 127.0.0.1 514
Solution 2: Use the recovery-mode approach and remove or alter the audit devices
If there is no other recourse and you must fix the issue consider the recovery-mode approach and removal or alteration of the audit devices as elaborated on the KB Article: [Recovery using recovery-mode] which should be repeated the same on all Vault instances in a cluster.
Outcome
After enabling the syslog local service at OS level, Vault operational logs will record successful messages and Vault will start accepting requests and commands.