Manually invoking the vault rewrap process fails to process all partially_seal_wrapped_paths

Avatar
Gerard Mansvelder
  • Updated

Introduction

In this particular scenario Vault Seal High Availability is enabled and was originally configured to use 2 seals.
One seal has been unavailable for some time and has been permanently removed from the Vault Configuration file.

Problem

Running vault read -format=json /sys/seal-backend-status returns:

{
  "request_id": "",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "backends": [
      {
        "healthy": true,
        "name": "awskms"
      }
    ],
    "fully_wrapped": false,
    "healthy": true
  },
  "warnings": null
}

Running vault read -field=seals -format=json sys/config/state/sanitized returns:

[
  {
    "disabled": false,
    "name": "awskms",
    "priority": 1,
    "type": "awskms"
  }
]

Please note that only one seal is currently enabled and that the seal is healthy "healthy": true and that fully_wrapped is false fully_wrapped": false

Running vault write -f sys/sealwrap/rewrap results in the following lines being displayed in the Vault Operational logs:

... [INFO]  seal.rewrap: seal re-wrap is running in the background
... [INFO]  seal.rewrap: seal re-wrap progress: entries processed=500
... [INFO]  seal.rewrap: seal re-wrap progress: entries processed=1000
... [ERROR] seal.rewrap: seal re-wrap failed: error while scanning entries: error="aborting seal re-wrap: partially-wrapped entries found: a seal has become unhealthy" entries processed=85404
... [INFO]  seal.rewrap: seal re-wrap completed: entry processing stats: succeeded=85404 failed=0 total=85404 successful=true

Prerequisites (if applicable)

  • Vault Enterprise 1.19.x or earlier versions
  • Seal High Availability enabled

Cause:

In this specific scenario running vault write -f sys/sealwrap/rewrap is not required, and the message below can ignored:

seal.rewrap: seal re-wrap failed: error while scanning entries: error="aborting seal re-wrap: partially-wrapped entries found: a seal has become unhealthy

Overview of possible solutions (if applicable)

Solutions:

After enabling the sys/raw endpoint, it is possible to list the partially_seal_wrapped_paths by executing: vault list sys/raw/partially_seal_wrapped_paths/ | wc -l

For example:

vault list sys/raw/partially_seal_wrapped_paths/ | wc -l
45818

The Vault Operational Log shows that rewrap process is still in progress:

... [TRACE] decrypted value using seal: seal_name=awskms
... [DEBUG] sealwrap: upgrading key entry: key=core/master
... [TRACE] sealwrap: wrapping entry: key=core/master
... [TRACE] encrypted value using seal: seal=awskms keyId=arn:aws:kms:eu-central-1:************:key/********-****-****-****-************
... [TRACE] successfully encrypted value: encryption seal wrappers=1 total enabled seal wrappers=1
... [DEBUG] sealwrap: unwrapping entry: key=core/keyring

Running vault list sys/raw/partially_seal_wrapped_paths/ | wc -l repeatedly shows that the number of entries in partially_seal_wrapped_paths is decreasing.

Outcome

After some time running vault read -format=json /sys/seal-backend-status should display fully_wrapped : true

Additional Information

Articles in this section

See more