Introduction
This knowledge base article provides guidance for transitioning a Vault cluster from PKCS#11 seal to Transit seal auto-unsealing using Seal HA (High Availability) functionality, covering both RAFT and Consul backend storage configurations.
Problem
Brief description of the problem to be solved
Organizations need to migrate their Vault clusters from PKCS#11 HSM sealing to Transit seal auto-unsealing to leverage Seal HA capabilities. Initial attempts at direct seal migration may fail with cryptographic errors, and the process differs between RAFT and Consul backend storage types.
Prerequisites
- Vault Enterprise (Seal HA requires Enterprise features)
- Functional Transit seal host with appropriate keys and policies configured
- Current Vault cluster using PKCS#11 seal
- Administrative access to modify Vault configuration
- Understanding of your storage backend type (RAFT vs Consul)
Cause
List of how to encounter this error and what error messages appear
This issue typically occurs when:
- Attempting direct seal migration instead of using Seal HA approach
- Misunderstanding that seal migration is needed when Seal HA is the correct solution
- Not following proper sequence for seal configuration changes
Common error messages encountered:
"while unwrapping, value has no key-id in common with currently healthy seals. Trying all healthy seals"
"post-unseal upgrade seal keys failed: failed to decrypt recovery key: error decrypting seal wrapped value"
"invalid ciphertext: no prefix"
-
"cannot make changes to vault while seal re-wrap is in progress"
(Consul backend)
Overview of Possible Solutions
The primary solution involves using Vault's Seal HA functionality rather than attempting seal migration. The process varies depending on your storage backend:
- Solution 1: Seal HA implementation for RAFT storage backend
- Solution 2: Modified Seal HA implementation for Consul storage backend
Solutions
Solution 1: Seal HA Implementation for RAFT Storage Backend
Step 1: Enable Multi-Seal on Standby Nodes
- On a standby node, ensure
enable_multiseal = true
is set in vault.hcl - Add Transit seal stanza (without migrate flag):
seal "transit" { address = "https://<transit-host>:8200" token = "<transit-token>" key_name = "<key-name>" mount_path = "transit/" tls_skip_verify = "true" disabled = "false" }
- Keep existing PKCS#11 seal with
disabled = "false"
- Restart Vault service:
systemctl restart vault
Step 2: Verify Multi-Seal Configuration
vault status vault read sys/seal-backend-status vault read sys/sealwrap/rewrap
Step 3: Repeat on All Standby Nodes
Complete steps 1-2 on each standby node before proceeding.
Step 4: Handle Active Leader
- On active node, execute:
vault operator step-down
- Wait for leadership election to complete
- Apply configuration changes to the former leader (now standby)
- Restart Vault service
Step 5: Add Second Transit Seal
Repeat the process to add a second Transit seal for additional redundancy.
Step 6: Remove PKCS#11 Seal
- Starting with standby nodes, remove PKCS#11 stanza from configuration
- Restart Vault services
- Use step-down process for the active leader
- Remove PKCS#11 from former active leader
Solution 2: Modified Seal HA Implementation for Consul Storage Backend
Key Differences for Consul Backend:
-
vault operator step-down
may not function the same way - Use service restart method for leadership changes
- Additional attention to seal rewrap completion
Step 1-3: Same as RAFT Solution
Follow the same multi-seal enablement process on standby nodes.
Step 4: Modified Leadership Handling
# Instead of step-down, restart the active leader's service systemctl restart vault # Wait 30-60 seconds for leadership election # Monitor rewrap completion: vault read sys/sealwrap/status
Step 5: Wait for Seal Rewrap Completion
Critical for Consul backend - ensure rewrap is complete before configuration changes:
vault read sys/sealwrap/status
Output should indicate rewrap is complete before proceeding.
Step 6-7: Continue with Additional Seals and PKCS#11 Removal
Follow same process as RAFT solution, ensuring rewrap completion at each step.
Outcome
Brief description of how to confirm the problem is solved
Success Indicators:
-
vault status
shows Transit seal type - Vault cluster unseals automatically without manual intervention
- All nodes show healthy status
- No seal rewrap operations in progress
Verification Commands:
vault status vault read sys/seal-backend-status vault read sys/sealwrap/status
If the problem persists:
- Verify Transit host connectivity and token permissions
- Check that all seal rewrap operations are complete before making changes
- For Consul backend, ensure leadership transitions complete before configuration modifications
- Review Vault logs for specific error details