Run a vault HA cluster on docker using dynamoDB storage backend

Avatar
Himanshu Sharma
  • Updated

Introduction

This guide covers the quick installation instruction to setup vault HA cluster inside docker containers using DynamoDB as storage backend. This setup will involve multiple Vault instances, a DynamoDB table for storing Vault's state, and configuring the HA mode for Vault.

Expected Outcome

This setup gives you a highly available Vault cluster using docker-compose having DynamoDB as the storage backend. The Vault instances will handle automatic failover, and since DynamoDB is highly available and scalable, you can achieve a resilient Vault deployment for production use.

Prerequisites

  • Docker & Docker Compose installed on your machine. In my lab setup:-

Docker version:- 27.2.0

Docker Compose version:- Docker Compose version v2.29.2-desktop.2

  • AWS DynamoDB table (for example, vault-ha) set up on AWS.

  • AWS credentials configured locally for Docker containers to access DynamoDB.

Use Case

Though this guide provides you quick and cost-effective setup method to run Vault HA using docker-compose with DynamoDB, however this guide holds values for production use-case too where Vault HA cluster would be running on non-ephemeral or persistent instance or could be on K8s cluster.

Procedure

  • Create a DynamoDB table:-

You need to create a DynamoDB table that Vault will use as the backend. Here is an example of how to do this in the AWS Console or using the AWS CLI:

aws dynamodb create-table \
  --table-name vault-ha \
  --attribute-definitions AttributeName=lock,AttributeType=S \
  --key-schema AttributeName=lock,KeyType=RANGE \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

  • Create a docker-compose file:-

The Docker Compose file will define multiple Vault instances running in HA mode, using DynamoDB as the storage backend.

#version: '3.8'

services:
  vault-1:
    image: hashicorp/vault:latest
    container_name: vault-1
    environment:
      - AWS_ACCESS_KEY_ID=<redacted>
      - AWS_SECRET_ACCESS_KEY=<redacted>
      - AWS_SESSION_TOKEN=<redacted>
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://vault-1:8200
      - VAULT_CLUSTER_ADDR=http://vault-1:8201
      - VAULT_LOG_LEVEL=debug
      - VAULT_UI=true
    volumes:
      - ./vault-config/vault.hcl:/vault/config/vault.hcl
      - vault-1-data:/vault/data
    ports:
      - "8200:8200"
      - "8201:8201"
    networks:
      - vault-net
    command: "server"
    #depends_on:
      #- vault-2
      #- vault-3
    restart: unless-stopped
    cap_add:
      - IPC_LOCK

  vault-2:
    image: hashicorp/vault:latest
    container_name: vault-2
    environment:
      - AWS_ACCESS_KEY_ID=<redacted>
      - AWS_SECRET_ACCESS_KEY=<redacted>
      - AWS_SESSION_TOKEN=<redacted>
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://vault-2:8200
      - VAULT_CLUSTER_ADDR=http://vault-2:8201
      - VAULT_LOG_LEVEL=debug
      - VAULT_UI=true
    volumes:
      - ./vault-config/vault.hcl:/vault/config/vault.hcl
      - vault-2-data:/vault/data
    ports:
      - "8202:8200"
      - "8203:8201"
    networks:
      - vault-net
    command: "server"
    #depends_on:
      #- vault-1
      #- vault-3
    restart: unless-stopped
    cap_add:
      - IPC_LOCK

  vault-3:
    image: hashicorp/vault:latest
    container_name: vault-3
    environment:
      - AWS_ACCESS_KEY_ID=<redacted>
      - AWS_SECRET_ACCESS_KEY=<redacted>
      - AWS_SESSION_TOKEN=<redacted>
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://vault-3:8200
      - VAULT_CLUSTER_ADDR=http://vault-3:8201
      - VAULT_LOG_LEVEL=debug
      - VAULT_UI=true
    volumes:
      - ./vault-config/vault.hcl:/vault/config/vault.hcl
      - vault-3-data:/vault/data
    ports:
      - "8204:8200"
      - "8205:8201"
    networks:
      - vault-net
    command: "server"
    #depends_on:
      #- vault-1
      #- vault-2
    restart: unless-stopped
    cap_add:
      - IPC_LOCK

networks:
  vault-net:
    driver: bridge

volumes:
  vault-1-data:
  vault-2-data:
  vault-3-data:

  • Vault configuration file (passed reference in docker-compose /vault-config/vault.hcl)

You will also need to create a configuration file vault.hcl for Vault to specify the storage backend and other settings. This will be referenced by each Vault instance in the docker-compose.yml.

# Enable DynamoDB storage backend
storage "dynamodb" {
  # DynamoDB table to store Vault's HA state
  table = "vault-ha"

  ha_enabled = "true"
  # Region where your DynamoDB table is located (e.g., us-east-1)
  region = "ap-south-1"

  # DynamoDB endpoint for local testing (uncomment if using DynamoDB Local)
  # endpoint = "http://localhost:8000"
}

# Enable listener on port 8200 (HTTP API)
listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable = 1  # Disable TLS for testing (set to 0 for production)
}

# Enable the Vault UI (optional)
ui = true

# Configure the Vault HA cluster
api_addr = "http://0.0.0.0:8200"
cluster_addr = "http://0.0.0.0:8201"

# Enable the Vault audit log (optional)
audit {
  file_path = "/vault/logs/audit.log"
}

  • Start the Vault cluster

Now you can start your Vault HA cluster using Docker Compose:

docker compose up

This will start three Vault instances running in HA mode, with DynamoDB as the storage backend. The Vault instances will automatically form a cluster using the VAULT_CLUSTER_ADDR and communicate with each other for high availability.

By exec to one of the vault docker container, check the status of vault using below commands.

/ # vault operator members 
Host Name       API Address            Cluster Address         Active Node    Version    Upgrade Version    Redundancy Zone    Last Echo
---------       -----------            ---------------         -----------    -------    ---------------    ---------------    ---------
c6889579ca79    http://vault-1:8200    https://vault-1:8201    true           1.15.5     n/a                n/a                n/a
3c3e114f4f84    http://vault-2:8200    https://vault-2:8201    false          1.15.5     n/a                n/a                2024-11-09T10:36:14Z
952371a40702    http://vault-3:8200    https://vault-3:8201    false          1.15.5     n/a                n/a                2024-11-09T10:36:15Z
/ # 
/ # 
/ # 
/ # vault status 
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.15.5
Build Date      2024-01-26T14:53:40Z
Storage Type    dynamodb
Cluster Name    vault-cluster-89d47aba
Cluster ID      92b64cd2-d68f-073b-07b8-d8258b192a0f
HA Enabled      true
HA Cluster      https://vault-1:8201
HA Mode         active
Active Since    2024-11-09T10:33:52.214880505Z
/ #

Additional Information

  • DynamoDB works differently from the integrated storage backend (raft) where in absence of quorum condition not matching with N/2+1 cluster goes down, however with DynamoDB it kept on running even if let’s say 2 out of 3 vault instance goes down.

/ # vault operator members 
Host Name       API Address            Cluster Address         Active Node    Version    Upgrade Version    Redundancy Zone    Last Echo
---------       -----------            ---------------         -----------    -------    ---------------    ---------------    ---------
952371a40702    http://vault-3:8200    https://vault-3:8201    true           1.15.5     n/a                n/a                n/a

Articles in this section

See more

Related articles