Introduction:
Vault known issue where significant increase in memory usage has been observed after an upgrade from older versions of Vault to 1.13.7+, 1.14.3+ or 1.15.0+
Brief Summary of the Issue:
- A memory leak was introduced to Vault in 1.13.7, 1.14.3 and 1.15.0 where requests triggering a policy check create a logger that is never removed. The side effect of this leak is unbounded consumption of memory until out-of-memory processes are triggered by the operating system.
- Vault is unexpectedly storing references to ephemeral sub-loggers which prevents them from being cleaned up, leading to a memory leak. This impacts many areas of Vault, but primarily logins in Enterprise.
- This memory leak is more prevalent in Vault Enterprise than Community Edition. Operators may experience increased memory usage after upgrading Vault to one of the affected versions above.
Affected versions:
This issue affects Vault Community and Enterprise versions:
- 1.13.7+ (1.13.7, 1.13.8, 1.13.9)
- 1.14.3+ (1.14.3, 1.14.4, 1.14.5)
- 1.15.0+ (1.15.0, 1.15.1)
It is highly recommended to hold off on upgrades to the affected versions until the fix is released.
Any Workarounds:
Engineering has released the fix in the following minor version releases; but a temporary workaround if you cannot upgrade, would be to restart the node experiencing the memory spike. If it is an active node; guidance is to perform vault operator step-down and then restart the node once a new node takes over the leadership.
Fix Version
-
Binaries:
- 1.13.10: 1.13.10 CE and 1.13.10 ENT
- 1.14.6: 1.14.6 CE and 1.14.6 ENT
- 1.15.2: 1.15.2 CE and 1.15.2 ENT
- Docker images
Additional Information: