Problem
After rebooting a node, the Raft database is taking 15-20minutes to start/open.
The following message can be seen in the Vault operational logs:
storage.raft.fsm: time to open database: elapsed=27m45.859645927s path=/data/vault.db
storage.raft.fsm: time to open database: elapsed=13m54.484803456s path=/data/vault.db
This delay does not happen when the Vault service itself is restarted, only when the server node is rebooted.
Prerequisites
- Vault Enterprise Version 1.8.2
Cause
- This issue is tracked under the following internal thread with our Engineering team:
VAULT-4030
Overview of possible solutions
Solution:
-
Setup the
VAULT_RAFT_FREELIST_SYNC=1
environment variable on the server, prior to the hard shutdown.
NOTE: Enabling this option will probably degrade write performance. That's the tradeoff: slower writes vs slower restart after hard shutdown.