Introduction
Problem
Utilizing PKI engine as an Intermediate CA for signing and issuing SSL certificates but when attempting to list certificates with a command such as:
certificates (e.g. curl -s -k -X LIST -H "X-Vault-Token: ${token}" https://localhost:8200/v1/pki_int_signed/certs)
you receive the following error(s):
{"errors": ["internal error"]}
-Or-
Error Ember Data Request GET /v1/pki_int_signed/certs returned a 500 Payload (application/json) [object Object] internal error.
Additionally you note the following errors in the logs:
Mar 22 1:46:38 vault-server vault[9414]: 2021-03-22T17:46:38.028+0100 [ERROR] audit: backend failed to log response: backend=syslog/ error="write unixgram @->/dev/log: write: message too long"
Mar 2217:46:38 vault-server vault[9414]: 2021-03-22T17:46:38.028+0100 [ERROR] core: failed to audit response: request_path=pki_int_signed/certs/ error="1 error occurred:
Mar 22 17:46:38 vault-server vault[9414]: * no audit backend succeeded in logging the response
Cause
In this case the log line exceeds the syslog size limitation, which is dependent on the implementation, or when using UDP mode, the UDP message size limitation which works out to 1024 bytes. Some syslog implementations also cap the TCP maximum message size at 8192 bytes as well.
Overview of possible solutions
Solutions:
Some recommendations here to deal with the long audit log lines:
Various implementations provide a way to increase message maximum size for TCP messages beyond the 8192 byte boundary.
- Use a syslog implementation in TCP mode with an increased maximum message size to handle the longer messages
- Use a socket based audit device to send log messages to a log aggregation tool instead of using syslog
- Use a file based audit device
- Use a combination of 2 and 3
Outcome
One way to test whether this is in fact the case is to enable a file based audit device and temporarily disable syslog and re-run the query. If successful you have confirmed you are encountered the error condition described above.
Additional Information
References:
https://www.vaultproject.io/docs/secrets/pki
https://www.vaultproject.io/docs/audit#enabling-disabling-audit-devices