The following errors, when identified in the Vault operational logs prevent the replication to establish. These logs were captured from the live environment of Vault backed up by Consul.
vault[75843]: [ERROR] core: unable to retrieve
replicated cluster
vault[75843]: [ERROR] replication: failed to invalidate key, suspending replication:
key=core/cluster/shared/filteredpaths/custom_name error="core: unable to retrieve replicated cluster"
vault[75843]: [ERROR] replication: encountered error, applying backoff:
backoff=2s error="core: unable to retrieve replicated cluster"
To overcome the issue, we first need to check on the path and the values stored inside said path via:
consul kv get vault/core/cluster/shared/filteredpaths/custom_name
In general, it has been observed that this happens because the DR Primary cluster used to be the Performance Replication (PR) Primary cluster, and the older data around `path filtering` still remains within the Vault.
If the cluster is not a PR cluster anymore, it would be safe to delete the data inside the `filteredpaths` via:
consul kv delete vault/core/cluster/shared/filteredpaths/custom_name
The following output will show that the KV was successfully deleted from Consul:
consul kv get -keys vault/core/cluster/
Once the keys are deleted, replication has to be established from scratch. Vault active/leader node has to be restarted if replication is still breaking after the key is deleted.