Introduction:
The expiration manager is an internal Vault component that’s not directly exposed to the user. It is the owner of the lease store, and performs the following functions:
- at startup, loads all lease entries from storage into memory
- when a lease is created or restored, starts a goroutine to handle revocation when TTL expires
- called when a lease is explicitly revoked before its TTL expires, to cleanup the lease entry
- periodically publishes num_leases metrics based on pending leases in memory
Revocation mechanics: Sync, Failures and Forcing
User revoke requests
By default revokes done using the revoke endpoints are lazy: they simply set the TTL to 0 and allow expiration manager to handle them as though they’d expired normally. There is an undocumented sync option which instead asks for them to be fully revoked while the caller waits.
To revoke the expiration manager routes a revoke request to the backend that originated the lease, using the path stored in the lease entry. This is what actually reaches out to external systems and revokes whatever we created there when the lease was generated. If the external revocation fails the caller gets an error unless they provided the force flag. When force is true, we ignore the external error and proceed to clean up the lease entry and secondary index if any.
Internal revoke on expiry
When a TTL reaches 0 and the expiration manager needs to revoke a lease, it does much the same as when explicitly requested to via a revoke endpoint. The main difference is failure handling: when an error occurs in the external system’s revocation, we log it in the system log, sleep for an exponential backoff, and retry. Currently there are 6 attempts made, after which we go ahead and delete the lease entry despite it not being revoked externally.
Performance standbys
Performance standbys also run the Expiration Manager to keep track of which leases are valid. In their case when expiration occurs the lease is simply removed from memory, since it’s the active node’s job to handle revocation and storage cleanup.
Performance
Leases are frequently involved in performance problems, e.g. restoring leases into memory at startup can do a huge amount of (read) I/O, or an accumulation of leases coming due to at the same time can lead to a storm of expirations that can swamp the storage layer. Several Vault releases have introduced changes to better handle these scenarios, along with a few options for further tuning the Expiration Manager behavior (if desired).
However, the default values are expected to be the best options, so please only tune these values if you know what you are doing (i.e; testing the effect of these changes in a test environment with comparable lease activity, adequate monitoring of lease-specfic metrics as well as base-level system metrics for measuring changes in performance, maintenance windows for account for unexpected service impact, up-to-date backups, etc). It is highly advisable to contact our Support team before changing any of these values to get a verification that changes are needed or will even help with your specific scenario.
In 1.6+, we introduced the environment variable VAULT_16_REVOKE_PERMITPOOL, which limits how many goroutines can be revoking at once. The default - when this is not set - is to attempt to revoke all expired leases simultaneously.
In 1.7+, we throttle the expirations automatically, and no user action is required to avoid swamping the storage layer. If further throttling is desired or required, the environment variable VAULT_LEASE_REVOCATION_WORKERS is set by default to 200, and can be reduced.
In 1.8+, we introduced enhanced expiration manager functionality to internally mark leases as irrevocable after 6 failed attempts at revocation. This provides a way to stop attempting revocation on leases which are identified as irrevocable. An HTTP API and CLI command are also available to assist operators in identifying irrevocable leases.