HA Auto-Unseal with dual transit clusters in air-gapped vault environments with secure token injection

Introduction:

While cloud KMS solutions are popular, multi-cloud flexibility is necessary for environments that demand more control. However, in a strict air-gapped environment, using Vault's Transit secret engine for auto-unseal offers a powerful alternative. This article delves into configuring Vault auto-unseal leveraging two separate, highly available (HA) Vault auto-unseal transit clusters to unseal the primary cluster.

The focus of this article is to demonstrate how to securely provide two transit tokens to vault cluster running with seal-HA without specifying the token in vault configuration.

Need for dual HA transit clusters:

• Achieving Redundancy and seal-HA in vault without any dependency on cloud KMS solutions.

Detailed configuration steps:

1. Set up Transit Cluster A (e.g., in Region 1) using the steps mentioned in official documentation.

2. Set up Transit Cluster B (e.g., in Region 2). Repeat all the steps as done in step 1.

3. Configure Primary Vault Cluster(s) for Dual Auto-Unseal

• example vault-primary.hcl:

   

  storage "raft" {
    path    = "/vault/data"
    node_id = "primary-vault-1"
  }
  
  listener "tcp" {
    address     = "0.0.0.0:8200"
    tls_disable = "false" # Recommended
    tls_cert_file = "<cert_path>"
    tls_key_file  = "<key_path>"
  }
  
  cluster_addr = "<cluster_address>:8201"
  api_addr     = "https://<primary-vault-load-balancer>:8200"
  enable_multiseal = true
  
  # Auto-Unseal using Transit Cluster A
  seal "transit" {
    address         = "https://<transit-a-load-balancer>:8200"
    priority   = "1"
    token           = "env://VAULT_TOKEN_first"
    disable_renewal = "false"
    key_name        = "unseal-key-a"
    mount_path      = "transit/"
    tls_skip_verify = "false" # Set to true only for testing, use proper CAs in prod
    # client_cert = "/path/to/client-cert.pem"
    # client_key  = "/path/to/client-key.pem"
    # ca_cert     = "/path/to/transit-a-ca.pem"
  }
  
  # Auto-Unseal using Transit Cluster B
  seal "transit" {
    address         = "https://<transit-b-load-balancer>:8200"
    priority   = "2"
    token           = "env://VAULT_TOKEN_second"
    disable_renewal = "false"
    key_name        = "<autounseal-keyname"
    mount_path      = "transit/"
    tls_skip_verify = "false" # Set to true only for testing, use proper CAs in prod
   
    # client_cert = "/path/to/client-cert.pem"
    # client_key  = "/path/to/client-key.pem"
    # ca_cert     = "/path/to/transit-b-ca.pem"
  }

  4. Securely Provide Transit Tokens to vault cluster: In the latest release of Vault, starting with Vault 1.19.0 and later, you can indeed use indirect value references in the transit seal stanza. This means you can specify environment variables, for example, by setting token = "env://VAULT_TOKEN_first" and token = "env://VAULT_TOKEN_second" . In this case, vault would expect to have values assigned to environment variables named VAULT_TOKEN_first and VAULT_TOKEN_second .

Reference:

• Seal HA
• Setup-Vault-with-Highly-available-Unseal-mechanisms