(Internal) Vault fails to unseal after upgrading to Vault 1.16.23, 1.18.12, 1.19.7 or 1.20.1 or newer

Avatar
Gerard Mansvelder
  • Updated

Introduction

Problem

Vault fails to unseal after upgrading to Vault 1.16.23, 1.18.12, 1.19.7 or 1.20.1 or newer if Vault Audit File Device mode configuration parameter contains the executable bit.

The Vault Operational Log shows:

[ERROR] core: failed to create audit entry: path=file/ error="unable to create new audit backend: file mode may not be executable: 744"
[INFO]  core: pre-seal teardown starting
[DEBUG] rotation-job-manager: rotation entries collected: num_existing=0
[INFO]  core: stopping raft active node
[INFO]  core: stopping replication
[TRACE] core.snapshotmgr: shutting down automatic snapshots
[INFO]  core: closed sync connection
[TRACE] replication.perf.logshipper: interrupting streams
[TRACE] replication.perf.logshipper: done interrupting streams
[TRACE] replication.dr.logshipper: interrupting streams
[TRACE] replication.dr.logshipper: done interrupting streams
[INFO]  core: replication stopped
[DEBUG] audit: removing reload function: path=file/
[INFO]  core: shutting down census manager
[DEBUG] reporting: census manager is not active
[DEBUG] core: census manager shutdown complete
[DEBUG] expiration: stop triggered
[TRACE] expiration.job-manager: terminating job manager...
[TRACE] expiration.job-manager: terminating dispatcher
[DEBUG] expiration: finished stopping
[DEBUG] rotation-job-manager: stopping rotation manager
[TRACE] rotation-job-manager: terminating job manager...
[TRACE] rotation-job-manager: terminating dispatcher
[DEBUG] rotation-job-manager: finished stopping rotation manager
[INFO]  rollback: stopping rollback manager
[DEBUG] system.secrets_sync: stopping secrets sync reconciliation
[DEBUG] system.secrets_sync: stopping secrets sync queue
[TRACE] system: terminating job manager...
[TRACE] system: terminating dispatcher
[DEBUG] sealwrap: stopping upgrades
[DEBUG] core: set storage to read-write
[INFO]  core: pre-seal teardown complete
[ERROR] core: post-unseal setup failed: error="failed to setup audit table"

Prerequisites (if applicable)

  • Vault Enterprise Vault 1.16.23, 1.18.12, 1.19.7 or 1.20.1 or newer
  • Only one Vault Audit File Device is enabled and has the mode configuration parameter specified containing the execute bit.

Cause

  • In Vault Enterprise 1.16.23, 1.18.12, 1.19.7 or 1.20.1 or newer the following issues have been addressed: 

    audit: privileged vault operator may execute code on the underlying host (CVE-2025-6000). Add additional verifications to the target of file audit sinks. [GH-31211,HCSEC-2025-14]

    Because of this the Vault Audit File Device mode configuration parameter is no longer allowed to contain the execute bit. 

    For example:

    vault audit enable file file_path=/opt/vault/vault-audit.log mode=744

    Please note the 7, this contains the execute bit.

    r (read): 4
    w (write): 2
    x (execute): 1
     

Overview of possible solutions (if applicable)

Solutions:

  • Downgrade Vault to a version prior to 1.16.23, 1.18.12, 1.19.7 or 1.20.1 with data store rollback and disable or reconfigure the Audit File device without using the execute bit prior to re-attempting the Upgrade. See Vault rollback upgrade

    For example:
     

    vault audit disable file
vault audit enable file file_path=/opt/vault/vault-audit.log mode=644

  • Vault Recovery mode can be leveraged to delete the affected Audit Device, please engage HashiCorp Global support for further assistance and prior to attempting to execute the steps, as this involves both using recovery mode as well as performing a quorum recovery.

    Recovery tokens

    Recovery tokens are issued in much the same way as root tokens are generated, only using a different endpoint. Unlike root tokens, the recovery token is not persisted, so if Vault is restarted into recovery mode a new one must be generated.

    Only a single recovery token can be generated. If lost, restart Vault and generate a new one.

    Raw requests

    Requests can be issued to sys/raw in just the same way as in regular Vault server mode. The only difference is that in recovery mode, X-Vault-Token must contain a recovery token instead of a service or batch token.

    Reform the raft cluster

    Recovery mode Vault automatically resizes the cluster to size 1. This is necessary because the Raft protocol won't allow changes to be made without a quorum, and in recovery mode we wish to make changes using a single node.

    This means that after having used recovery mode, part of the procedure for returning to active service must include re-forming the raft cluster. There are two ways to do so: either delete the vault data directory on the other nodes and re-join them to the recovered node, or use the Manual Recovery Using peers.json approach to get all nodes to agree on what nodes are part of the cluster.
     

    Prior to using the recovery mode, please make sure to have a backup of the directories containing the RAFT database files.


    Please find an example of the relevant steps below:

#Start a single instance of Vault in Recovery mode
vault server -config=/etc/vault.d/vault.hcl -recovery

#Using another terminal please execute the steps below:
vault operator generate-root -recovery-token -init
vault operator generate-root -recovery-token (provide the recovery or shamir keys)
vault operator generate-root -recovery-token -decode=xxxxxxxx -otp=xxxxxxxxxx 
export VAULT_TOKEN= #Use the generated Recovery Token

vault read sys/raw/core/audit
Key      Value
---      -----
value    {"type":"audit","entries":[{"table":"audit","path":"file/","type":"file","description":"","uuid":"5946a965-b5f6-6361-f905-fc23d9bbe57e","backend_aware_uuid":"","accessor":"audit_file_085559ec","config":{},"options":{"file_path":"/opt/vault/vault-audit.log","mode":"744"},"local":false,"seal_wrap":false,"namespace_id":"root"}]}

vault delete sys/raw/core/audit

After performing the above steps, please make sure to stop Vault to exit the recovery mode and to create a peers.json file to perform a quorum recovery.

Additional Information

Articles in this section

See more