Introduction
There are scenarios on which the sealing mechanism must be migrated from TRANSIT to a different KMS cloud provider or SHAMIR keys. Hashicorp Vault is offering multiple KMS cloud option. This article is covering the seal migration from TRANSIT to AWS KMS.
Prerequisites
- Working AWS KMS with sufficient rights & verified access using aws-cli tools.
Use Case
Migrating may be performed directly from a transit auto-unseal mechanism to an awskms
auto-unseal mechanism, without an intermediate Shamir seal stage. Please bare in mind that the success of performing the migration in one step is subject to the target KMS seal working in terms of access / network connectivity as well as sufficient rights permissions and roles to use the KMS in question.
Supported: transit -> awskms
Procedure
-
Backup the current Vault data and configuration according to Vault SOP.
-
Identify the standby (follower) nodes and the leader node(active).
As for example for a 3 node Vault cluster might look like:vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-us-east-2-us-east-2a-test 192.168.0.5:8201 leader true
vault-us-east-2-us-east-2b-test 192.168.0.21:8201 follower true
vault-us-east-2-us-east-2c-test 192.168.0.37:8201 follower true -
Stop Vault process on last follower node (vault-us-east-2-us-east-2c-test) sudo systemctl stop vault
-
Modify the configuration of Vault (on this example it should reside on
/etc/vault.d/vault.hcl
) and adding the new AWS stanza like in the example from below where999999-ffff-4444-aaaa-a585c5138ba1
is a AWS Kms_key_id:seal "awskms" {
region = "us-east-2"
kms_key_id = "999999-ffff-4444-aaaa-a585c5138ba1"
} -
Add the parameter
disabled = "true"
into TRANSIT stanza as example:
( Please note that the token from this transit stanzatoken ="hvs.CAESIGeFD7mT_mFbhmszy5qMXEwbGFJZW1tNFNudEE1ZTlRb0gwYUk"
is the token required to authenticate to Vault server.)seal "transit" {
address = "http://ec2-329837.us-east-2.compute.amazonaws.com:8200"
disable_renewal = "false"
key_name = "autounseal"
mount_path = "transit/"
tls_skip_verify = "true"
# Tranzit TOKEN
token ="hvs.CAESIGeFD7mT_mFbhmszy5qMXEwbGFJZW1tNFNudEE1ZTlRb0gwYUk"
disabled = "true"
} -
Start Vault service
sudo systemctl start vault -
After editing the seal stanza and restarting Vault, the following log will be in the Vault operational logs. This confirms that Vault can be migrated from TRANSIT to
awskms
.
As example:... vault[3776]: 2023-11-06T18:41:37.721+0100 ... [INFO] core: post-unseal setup complete
-
To perform the migration, run the command
vault operator unseal -migrate
and enter the recovery keys from (TRANSIT Initialization). Repeat this until we meet the required key threshold. After the threshold is met, the migration is complete, and the seal type is now AWS.# Example:
echo "
Recovery Key 1: kLKgkcRpj7iFnUgr4H8fK2my5VSB22tS25yMlqheFlR4
Recovery Key 2: yU+cQObXbKT5NR9gh6+/ThGjaJN2F/GCD2q+pLLZxACA
Recovery Key 3: PbtoI7GEnjdVA014ye9xWsXkcJGAs9A+Opw+Xj+7XTTw"| \
awk '{system("vault operator unseal "$NF)}'(the output may look like):
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed true
Total Recovery Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce c0c0bef5-5838-5b50-de0c-62ec28d67d45
Version 1.12.2+ent
Build Date 2022-11-23T21:33:30Z
Storage Type raft
HA Enabled true
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed true
Total Recovery Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce c0c0bef5-5838-5b50-de0c-62ec28d67d45
Version 1.12.2+ent
Build Date 2022-11-23T21:33:30Z
Storage Type raft
HA Enabled true
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.12.2+ent
Build Date 2022-11-23T21:33:30Z
Storage Type raft
Cluster Name cluster-us-east-2
Cluster ID 9c8e059b-ea85-0981-ccfd-496bfa6d5723
HA Enabled true
HA Cluster https://192.168.0.5:8201
HA Mode standby
Active Node Address https://192.168.0.5:8200
Raft Committed Index 1183
Raft Applied Index 1182 -
Observe that after last unseal migrate execution command the Sealed value should be FALSE:
Sealed false -
Perform the same operation on the second follower(vault-us-east-2-us-east-2b-test) from points from 3 until 9.
-
In order to finalize the migration of the auto-unseal perform a
vault operation step-down operation
on the leader node, observe that the leadership is transferred to one of the follower nodes. - Stop the leader node, add the AWS stanza, and remove completely the TRANSIT stanza.
# AWS KMS auto unseal
seal "awskms" {
region = "us-east-2"
kms_key_id = "999999-ffff-4444-aaaa-a585c5138ba1"
} - Start the Vault service and observe that the auto-unseal is working as expected and that Vault is in unsealed state.
- Validate that the seal mechanism is the desired one by login to the Vault on the newly elected leader node and execute the command:
vault read -field=seals -format=json sys/config/state/sanitized
(the output may look like:)
[
{
"disabled": false,
"type": "awskms"
}
{
"disabled": true,
"type": "transit"
}
] - On the current leader node, execute a
vault operator step-down
to change the Vault leader node to the initial leader nodevault-us-east-2-us-east-2a-test
) that does not contain any TRANSIT stanza. - Stop Vault the first follower node (vault-us-east-2-us-east-2b-test) and remove completely the TRANSIT stanza. For example:
#seal "transit" {
# address = "http://ec2-329837.us-east-2.compute.amazonaws.com:8200"
# disable_renewal = "false"
# key_name = "autounseal"
# mount_path = "transit/"
# tls_skip_verify = "true"
# Tranzit TOKEN
# token ="hvs.CAESIGeFD7mT_mFbhmszy5qMXEwbGFJZW1tNFNudEE1ZTlRb0gwYUk"
# disabled = "true"
#} - Start Vault on the follower and validate if the auto-unseal mechanism is functioning OK and that the Vault is unsealed.
- Perform the same steps from 16 to 17 for the other follower node (vault-us-east-2-us-east-2c-test).
- Validate that the auto-unseal mechanism is the desired one and that there is no longer a migration in progress by login into the Vault on first node and executing the command:
vault read -field=seals -format=json sys/config/state/sanitized
[
{
"disabled": false,
"type": "awskms"
}
]
Additional Information
- Documentation - Vault Seal migration
- Documentation - Transit Seal stanza
- KB article - Seal migration KB article
- KB article - Identify seal type at runtime