Introduction
This KB article addresses a specific Vault operational issue encountered during recovery from a lost Raft quorum using the peers.json method. The error relates to a missing plugin metadata file, which prevents Vault from completing its post-unseal setup.
Issue Summary
After a Vault cluster lost quorum and recovery was attempted using the peers.json method, Vault failed to complete the post-unseal setup. The operational logs displayed an error regarding the inability to verify the vault-plugin-database-oracle plugin due to a missing metadata.json file.
Observed Logs
The following logs were observed in the Vault operational logs:
[INFO] storage.raft: raft recovery initiated: recovery_file=peers.json [INFO] storage.raft: raft recovery deleted peers.json [ERROR] core: post-unseal setup failed: error="failed to verify plugin \"vault-plugin-database-oracle\" version \"v0.11.0+ent\": failed to read metadata: open /opt/vault/plugins/vault-plugin-database-oracle_0.11.0+ent_linux_amd64/metadata.json: no such file or directory" [INFO] core: pre-seal teardown starting [INFO] core: stopping raft active node [TRACE] core.snapshotmgr: shutting down automatic snapshots [INFO] core: closed sync connection
Root Cause
The root cause of this issue is that Vault, during the post-unseal phase, attempts to verify the installed plugins by reading their metadata files. In this case, the required metadata.json file for the vault-plugin-database-oracle plugin (version v0.11.0+ent) is missing from the expected directory (/opt/vault/plugins/vault-plugin-database-oracle_0.11.0+ent_linux_amd64/). Without this file, Vault cannot validate the plugin, leading to a failure in the post-unseal setup process.
Workaround/Solution
To resolve this issue and allow Vault to complete the post-unseal setup:
- Ensure that the plugin binary and its corresponding
metadata.jsonfiles are present in the plugin directory specified in your Vault configuration. - If the
metadata.jsonfile is missing, you can regenerate it by re-registering the plugin using thevault plugin registercommand, or by reinstalling the plugin following the official download release for plugin management. - After restoring or registering the plugin and its metadata, restart the Vault service and proceed with the unseal process.
References