How to Setup Vault with Highly available Unseal mechanisms? - Seal HA

Introduction:

To provide resiliency against a seal service/mechanism outage, Seal High Availability allows you to specify a minimum of two auto-seals and a maximum of three. Additional seals must be configured in Vault's server configuration file to use Seal HA.

Important: Seal HA is still an Enterprise-Beta feature and hence it's not recommended to be implemented for production workloads.

Prerequisites:

1. Vault Enterprise version 1.15.0+ent or higher.
2. AWS KMS keys with appropriate permission.

Use Case:

Two autounseal mechanisms are to be used in the Vault configuration. This could be a combination of transit/HSM with CSP keys or both seals can be from CSPs. Here demonstration uses two AWS KMS keys.

Steps:

• By default seal HA is not enabled, So please enable it by using the environment variable below:

VAULT_ENABLE_SEAL_HA_BETA=true 

• vault service/unit file after setting up variable:

• Verify if seal-HA is enabled or not in vault operation logs:

1. First, Initialize the vault with a single seal mechanism( if the Vault is not running currently with one seal mechanism).
2. Once the vault is initialized, and up and running, add a stanza for the second seal mechanism.
3. Perform a vault service restart.

• Example config file having seal-HA:

storage "raft" {
path = "/storage"
node_id = "tester"
}

listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = "true"
}

seal "transit" {
address = "Auto-unseal server ip address:8200"
mount_path = "transit/"
tls_skip_verify = "true"
key_name = "autounseal"
token = "Transit token value"
priority ="2"
}

seal "awskms" {
region     = "us-east-1"
kms_key_id = "6c390016-b20b-xxx-xxx-eb4xxe9bde0"
priority = "1"
access_key = "<Access-key>"
secret_key = "<secret-key"
}

disable_mlock = true
api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true

Note: Priority is mandatory if more than one seal is specified. Priority tells Vault the order in which to try seals during unseal (least priority first), in the case more than one seal can unwrap a seal-wrapped value, the order in which to attempt decryption, and which order to attempt to source entropy for entropy augmentation. This can be useful if your seals have different performance or cost characteristics.

• Sample operational logs when all seal-mechanisms are healthy:

Dec 09 15:29:06 tester vault[5480]: 2023-12-09T15:29:06.820+0530 [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
Dec 09 15:29:06 tester vault[5480]: 2023-12-09T15:29:06.861+0530 [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=transit

• If one of the seal mechanisms is down :

Dec 09 15:34:24 tester vault[5480]: 2023-12-09T15:34:24.699+0530 [WARN]  error encrypting with seal: seal=transit
Dec 09 15:34:59 tester vault[5480]: 2023-12-09T15:34:59.794+0530 [DEBUG] sealwrap: skipping rewrap of partially wrapped values, not all seals are healthy

Reference: 

ha-seal