Enterprise Only – These endpoints require Vault Enterprise.
Reindex is usually only required if you're running into a bug not just for a general catch-up. General "WALS are behind, need to catch up" is handled by Merkle sync automatically
The Benefit of Reindex is to keep the replicated data consistent across the primary and secondary, Vault maintains Merkle trees. If replication is in a bad state or data has been removed from the storage backend without Vault's knowledge, you can trigger reindexing of the Merkle tree via the /sys/replication/reindex. This endpoint reindexes the local data storage. This can cause a very long delay depending on the number and size of objects in the data store.
If we see this error message on the logs it means you have to reindex the primary cluster. Reindex Replication
[ERROR] replication: encountered error, applying backoff: backoff=2s error="state is still irreconcilable after reindex, try reindexing primary cluster"
Replication Reindex Process:
This outlines how the Vault Enterprise Replication Reindex function works and its tradeoffs.
This is true as of 0.11.x, step 2 was added in 1.1.3
The steps that the algorithm take are:
- Cache the last seen WAL index for the tree
- Copy over hashes of filtered data to the newly created Merkle tree
- Scan all data in storage
- Build a new tree based on this data
- Replay WALs from the cached last seen index (from step 1) to the latest WAL
- Lock tree
- Replay WALs from the end of the previous replay to the latest WAL
- Diff new tree and existing tree, replace any page that differs
- Flush all dirty pages of the tree to disk
- Unlock the tree
The reindexing process spends the majority of the time on Steps 1-4. During these steps, the process is fully in the background and Vault can service both read and write requests (assuming the node in question can service requests). Step 5 does a WAL replay to make sure we have included all updates that happened since the start of the data scan (index gathered in step 1).
Steps 6-9 operate while the tree is locked. This is so we can safely make the switch over to the newly created tree. During these steps Vault can service read requests only; write requests will block until the reindex is over. Step 6 does an additional WAL replay to ensure we have captured all updates that happened between the last replay and the time that we locked the tree. Step 8 flushes any dirtied tree pages to disk. The performance of this is based on how many pages are actively dirty which is a combination of how many pages were dirtied normally plus any pages we repaired during the reindex. The number of pages that are dirty prior to the reindex can be found by searching for a debug log containing the string “saved checkpoint: num_dirty=”
Step 9 unlocks the tree and finishes the reindex. Once unlocked Vault can again handle both read and write requests. All pending writes will be serviced.