There are times when investigations into leases using standard methods result in insufficient information to address issues that can occur with larger-scale Vault clusters that utilize massive amounts of leases. To better identify sources of out-of-control lease generation, or to locate overly-large collections of long-running leases, the Hashicorp Vault Support team has an internally-used tool that can read leases directly from the Vault API and generate reports based on what it finds.
Prerequisites (if applicable)
- Sample Leases Tool (acquired from the Vault Support team)
- The latest version of Go from golang.org installed on the machine running the tool
- A Vault token that is configured to provide access to the following paths:
- list sys/namespaces
- read sys/leases/lookup
- list sys/leases/lookup/* (requires sudo priv)
The Sample Leases tool can be run against the Vault server to generate a report with details about leases on the system, with a configurable sample size, broken down by namespace. This is very useful for identifying overly-large collections of leases that exist under a specific namespace or secrets engine, which have accumulated due to overly generous TTLs or due to an application or service using less-than-optimal patterns for generating secrets. After identifying locations of these large collections of leases, further steps can be taken to clean up excessive leases on the system or address usage patterns (by adhering to best practices and/or configuring rate-limiting quotas for secret generation.).
Ensure the prerequisites at the beginning of this article are present
- When requesting the tool from a member of the Support team, please reference the `vault-tools` repository and the tool name "sampleLeases"
- Install the latest version of Go from golang.org
- Set the VAULT_ADDR env variable to the appropriate API address for the Vault server in use
- Set the VAULT_TOKEN env variable to the token previously created with appropriate permissions
- Extract the tool (received from Vault Support) and run (in that same directory): `go run main.go`
- The tool will result in the following files:
- samples.csv: contains all leases read, with the following fields: namespace, prefix, suffix, creationTime, expireTime
- summary-creation.csv: contains quantile summaries: namespace, prefix, totalLeases, sampledLeases, 25th, 50th, 75th, 90th, 99th
- summary-expiry.csv: contains quantile summaries: namespace, prefix, totalLeases, sampledLeases, 25th, 50th, 75th, 90th, 99th
- After viewing these reports, you should be able to make a determination on where to focus efforts