How to safely disable Azure Key Vault Key Versions when using Vault Auto-unseal

Avatar
Daniel Arlington
  • Updated

Introduction:

There maybe a need to disable older Key Versions in AKV due to company security policy. This KB will show you how to safely disable the older Key Version in AKV.

 

Use Case:

In case a customer needs to disable an old or expired key version in Azure Key Vault. 

I used a three-node cluster to test the outcomes of rotating and disabling the AKV Key Version. I tested the effects of performing a “Leader Step-down” vs “Restarting All the Vault nodes”

A. Leader Step-down 

B. Restarting All the Vault nodes

 

Pre-requisites:

Please follow the tutorial on how to setup Auto-Unseal using Azure Key Vault here [1]

On an existing unsealed Vault using "shamir" you can add the following un-seal stanza to enable Azure Key Vault method of unsealing vault. You will still need the "Shamir" set of unseal keys and then run the operator unseal migrate command.

seal "azurekeyvault" {
  tenant_id     = "t63d9f94-0647-xxxx-xxxx-xxxxxxxxx"
  client_id     = "a29a8583-xxxx-xxxx-xxxx-xxxxxxxxx"
  client_secret = "xxxxxxxxxxxxxxxx"
  vault_name    = "learn-vault-xxxx"
  key_name      = "vault-keys-1234"
}

 

A. Leader Step-down (not recommended)

I performed the following steps.

  1. Rotated the AKV Key Version
  2. Performed a Leader Step-down
    $vault operator step-down
  3. We see in the logs that the new Key Version gets picked up.
  4. We disabled the old Key Version.
  5. Vault continues to work. However, there was a problem. As soon as we did another leader step down or restarted Vault it started to complain about the Key being disabled. This is due to the other Vault nodes not knowing the Key Version change.
  6. All the Vault nodes went into a Sealed State essentially locking you out of Vault.

To unseal Vault and start serving traffic, we had to re-enable the old Key Version and restart all the Vault nodes.
 

B. Restarting All the Vault nodes (recommended)

I performed the following steps.

  1. Rotated the AKV Key Version

  2. Performed a Vault Restart 

    a. Start with the Follower Nodes and then restart the Leader node.

  3. We see in the logs that the new Key Version gets picked up in each node when it get Unsealed.
  4. We disabled the old Key Version.
  5. We tried multiple "vault operator step-down" on the leader nodes. This worked fine without any issues observed.

So, our recommendation would be to ensure that all the nodes in the Vault Cluster need to be restarted.

The important point is that each Node must undergo a “Seal” and then “Unseal” process so that the old Key Version is used to unseal and then the new Key Version is updated on each node. Once each node has the New Key Version updated, you can disable the old Key Version.


 

References:

[1] https://developer.hashicorp.com/vault/tutorials/auto-unseal/autounseal-azure-keyvault

Articles in this section

See more