Spike in vault_core_in_flight_requests as a result of listing Auth Methods on Vault UI

Overview

In environments with large numbers of Vault authentication methods, listing these methods via Vault UI can trigger spikes in the vault_core_in_flight_requests metric. This occurs due to capabilities checks performed for each auth method during the rendering process.

Symptoms

• An increase in vault_core_in_flight_requests when accessing the Auth Methods page via Vault UI.
• Monitoring tools show a flurry of POST requests to /v1/sys/capabilities-self endpoint.

Cause

The Vault UI relies on capabilities checks to determine user permissions for each auth method. Up until version 1.21,  Vault's implementation used a data management library that issued individual requests for each resource, leading to a 1:1 coupling between auth methods and capabilities calls. 

The issue is more pronounced with Kubernetes authentication methods, which require checks at both root and configuration paths.

Resolution

Starting on Vault version 1.21, capabilities checks are batched, reducing the number of requests to a single call for all items rendered in the Auth Methods list. 

Additional Information

https://developer.hashicorp.com/vault/docs/internals/telemetry/metrics/all#vault-core-in_flight_requests