Introduction
Anything that connects and authenticates to Vault to accomplish a task is a client. For example, a user logging into a cluster to manage policies or a machine-based system (application or cloud service) requesting a database token are both considered clients. Vault client calculation and sizing can be complex to compute when measuring Vault Enterprise usage. Vault v1.6 introduced Vault Metrics dashboard in Web UI. This enhancement helps to find active client count and historical client counts within Vault.
Problem
When accessing the Vault Client Count in the Web UI the following error is observed (example below using Vault v1.15.6+ent):
When accessing the Vault Client Count API endpoint the same error code is observed.
The CLI Command used to query Vault Client Count API endpoint:
vault read -format=json sys/internal/counters/activity| jq -r ".data.total"
The resultant output from above CLI Command:
Error reading sys/internal/counters/activity: Error making API request.
URL: GET https://127.0.0.1:8200/v1/sys/internal/counters/activity
Code: 500. Errors:
* internal error
Cause
In this specific case the Vault Enterprise cluster only had a single Audit Device enabled. The Audit Device was of type syslog.
The following was observed in the Vault Operational logs:
{"@level":"error","@message":"failed to audit response","@module":"core","@timestamp":"2024-05-02T14:25:47.094393Z","error":"2 errors occurred:\n\t* event not processed by enough 'sink' nodes\n\t* event.(SyslogSink).Process: error writing to syslog: write unixgram @-\u003e/dev/log: write: message too long\n\n","request_path":"sys/internal/counters/activity"}
The above error is indicative of the Audit Device failing to audit the response to the request made to Vault. When Vault is unable to audit both the request and response, the request made to Vault will fail.
Solution
In order to prevent failure in the auditing of Vault requests and responses, multiple Audit Devices were enabled, and underlying cause of the failure of the syslog Audit Device was also addressed.
Outcome
Both the Web UI and Vault API / CLI requests successfully returned the Vault Client Count without error.
Additional Resources
- Vault Documentation: API Client Count
- Vault Documentation: Vault Clients & Entities
- Vault Documentation: FAQ Client Count
- Vault Documentation: Client Count Improvements (Vault 1.9.0 Release Notes)
- Vault Documentation: Audit Devices
- Vault Tutorial: Vault Usage Metrics