Vault known issue where significant increase in memory usage has been observed after an upgrade from older versions of Vault to 1.13.7+, 1.14.3+ or 1.15.0+
Brief Summary of the Issue:
- A memory leak was introduced to Vault in 1.13.7, 1.14.3 and 1.15.0 where requests triggering a policy check create a logger that is never removed. The side effect of this leak is unbounded consumption of memory until out-of-memory processes are triggered by the operating system.
- Vault is unexpectedly storing references to ephemeral sub-loggers which prevents them from being cleaned up, leading to a memory leak. This impacts many areas of Vault, but primarily logins in Enterprise.
- This memory leak is more prevalent in Vault Enterprise than Community Edition. Operators may experience increased memory usage after upgrading Vault to one of the affected versions above.
This issue affects Vault Community and Enterprise versions:
- 1.13.7+ (1.13.7, 1.13.8, 1.13.9)
- 1.14.3+ (1.14.3, 1.14.4, 1.14.5)
- 1.15.0+ (1.15.0, 1.15.1)
It is highly recommended to hold off on upgrades to the affected versions until the fix is released.
Engineering has released the fix in the following minor version releases; but a temporary workaround if you cannot upgrade, would be to restart the node experiencing the memory spike. If it is an active node; guidance is to perform vault operator step-down and then restart the node once a new node takes over the leadership.
- Docker images