Auto-unseal migration from Transit to AWS KMS

Avatar
Florin Țiucra-Popa
  • Updated

Introduction

There are scenarios on which the sealing mechanism must be migrated from TRANSIT to a different KMS cloud provider or SHAMIR keys. Hashicorp Vault is offering multiple KMS cloud option. This article is covering the seal migration from TRANSIT to AWS KMS.

Prerequisites

  • Working AWS KMS with sufficient rights & verified access using aws-cli tools.

Use Case

Migrating may be performed directly from a transit auto-unseal mechanism to an awskmsauto-unseal mechanism, without an intermediate Shamir seal stage. Please bare in mind that the success of performing the migration in one step is subject to the target KMS seal working in terms of access / network connectivity as well as sufficient rights permissions and roles to use the KMS in question.

Supported: transit -> awskms

Procedure

  1. Backup the current Vault data and configuration according to Vault SOP.

  2. Identify the standby (follower) nodes and the leader node(active).
    As for example for a 3 node Vault cluster might look like:

    vault operator raft list-peers
    Node Address State Voter
    ---- ------- ----- -----
    vault-us-east-2-us-east-2a-test 192.168.0.5:8201 leader true
    vault-us-east-2-us-east-2b-test 192.168.0.21:8201 follower true
    vault-us-east-2-us-east-2c-test 192.168.0.37:8201 follower true

  3. Stop Vault process on last follower node (vault-us-east-2-us-east-2c-test) sudo systemctl stop vault

  4. Modify the configuration of Vault (on this example it should reside on /etc/vault.d/vault.hcl) and adding the new AWS stanza like in the example from below where 999999-ffff-4444-aaaa-a585c5138ba1 is a AWS Kms_key_id:

    seal "awskms" {
      region = "us-east-2"
      kms_key_id = "999999-ffff-4444-aaaa-a585c5138ba1"
    }

  5. Add the parameter disabled = "true" into TRANSIT stanza as example:
    ( Please note that the token from this transit stanza token ="hvs.CAESIGeFD7mT_mFbhmszy5qMXEwbGFJZW1tNFNudEE1ZTlRb0gwYUk" is the token required to authenticate to Vault server.)

    seal "transit" {
      address = "http://ec2-329837.us-east-2.compute.amazonaws.com:8200"
      disable_renewal = "false"
      key_name = "autounseal"
      mount_path = "transit/"
      tls_skip_verify = "true"
      # Tranzit TOKEN
      token ="hvs.CAESIGeFD7mT_mFbhmszy5qMXEwbGFJZW1tNFNudEE1ZTlRb0gwYUk"
      disabled = "true"
    }

  6. Start Vault service
    sudo systemctl start vault

  7. After editing the seal stanza and restarting Vault, the following log will be in the Vault operational logs. This confirms that Vault can be migrated from TRANSIT to awskms.
    As example:

    ... vault[3776]: 2023-11-06T18:41:37.721+0100 ... [INFO] core: post-unseal setup complete

  8. To perform the migration, run the command
    vault operator unseal -migrate
    and enter the recovery keys from (TRANSIT Initialization). Repeat this until we meet the required key threshold. After the threshold is met, the migration is complete, and the seal type is now AWS. 

    # Example:
    echo "
    Recovery Key 1: kLKgkcRpj7iFnUgr4H8fK2my5VSB22tS25yMlqheFlR4
    Recovery Key 2: yU+cQObXbKT5NR9gh6+/ThGjaJN2F/GCD2q+pLLZxACA
    Recovery Key 3: PbtoI7GEnjdVA014ye9xWsXkcJGAs9A+Opw+Xj+7XTTw"| \
    awk '{system("vault operator unseal "$NF)}'

    (the output may look like):

    Key Value
    --- -----
    Recovery Seal Type shamir
    Initialized true
    Sealed true
    Total Recovery Shares 5
    Threshold 3
    Unseal Progress 1/3
    Unseal Nonce c0c0bef5-5838-5b50-de0c-62ec28d67d45
    Version 1.12.2+ent
    Build Date 2022-11-23T21:33:30Z
    Storage Type raft
    HA Enabled true
    Key Value
    --- -----
    Recovery Seal Type shamir
    Initialized true
    Sealed true
    Total Recovery Shares 5
    Threshold 3
    Unseal Progress 2/3
    Unseal Nonce c0c0bef5-5838-5b50-de0c-62ec28d67d45
    Version 1.12.2+ent
    Build Date 2022-11-23T21:33:30Z
    Storage Type raft
    HA Enabled true
    Key Value
    --- -----
    Recovery Seal Type shamir
    Initialized true
    Sealed false
    Total Recovery Shares 5
    Threshold 3
    Version 1.12.2+ent
    Build Date 2022-11-23T21:33:30Z
    Storage Type raft
    Cluster Name cluster-us-east-2
    Cluster ID 9c8e059b-ea85-0981-ccfd-496bfa6d5723
    HA Enabled true
    HA Cluster https://192.168.0.5:8201
    HA Mode standby
    Active Node Address https://192.168.0.5:8200
    Raft Committed Index 1183
    Raft Applied Index 1182

  9. Observe that after last unseal migrate execution command the Sealed value should be FALSE:
    Sealed false

  10. Perform the same operation on the second follower(vault-us-east-2-us-east-2b-test) from points from 3 until 9.

  11. In order to finalize the migration of the auto-unseal perform a vault operation step-down operation on the leader node, observe that the leadership is transferred to one of the follower nodes.

  12. Stop the leader node, add the AWS stanza, and remove completely the TRANSIT stanza.
    # AWS KMS auto unseal
    seal "awskms" {
      region = "us-east-2"
      kms_key_id = "999999-ffff-4444-aaaa-a585c5138ba1"
    }
  13. Start the Vault service and observe that the auto-unseal is working as expected and that Vault is in unsealed state.
  14. Validate that the seal mechanism is the desired one by login to the Vault on the newly elected leader node and execute the command:
    vault read -field=seals -format=json sys/config/state/sanitized

    (the output may look like:)

    [
     {
      "disabled": false,
      "type": "awskms"
     }
     {
      "disabled": true,
      "type": "transit"
     }
    ]
  15. On the current leader node, execute a vault operator step-down to change the Vault leader node to the initial leader node vault-us-east-2-us-east-2a-test) that does not contain any TRANSIT stanza.
  16. Stop Vault the first follower node (vault-us-east-2-us-east-2b-test) and remove completely the TRANSIT stanza. For example:
    #seal "transit" {
    #  address = "http://ec2-329837.us-east-2.compute.amazonaws.com:8200"
    #  disable_renewal = "false"
    #  key_name = "autounseal"
    #  mount_path = "transit/"
    #  tls_skip_verify = "true"
    #  Tranzit TOKEN
    #  token ="hvs.CAESIGeFD7mT_mFbhmszy5qMXEwbGFJZW1tNFNudEE1ZTlRb0gwYUk"
    #  disabled = "true"
    #}
  17. Start Vault on the follower and validate if the auto-unseal mechanism is functioning OK and that the Vault is unsealed.
  18. Perform the same steps from 16 to 17 for the other follower node (vault-us-east-2-us-east-2c-test).
  19. Validate that the auto-unseal mechanism is the desired one and that there is no longer a migration in progress by login into the Vault on first node and executing the command:
    vault read -field=seals -format=json sys/config/state/sanitized
    The output may look like:
    [
     {
      "disabled": false,
      "type": "awskms"
     }
    ]

Additional Information

Articles in this section

See more

Related articles