Introduction
This guide covers the quick installation instruction to setup vault HA cluster inside docker containers using DynamoDB as storage backend. This setup will involve multiple Vault instances, a DynamoDB table for storing Vault's state, and configuring the HA mode for Vault.
Expected Outcome
This setup gives you a highly available Vault cluster using docker-compose having DynamoDB as the storage backend. The Vault instances will handle automatic failover, and since DynamoDB is highly available and scalable, you can achieve a resilient Vault deployment for production use.
Prerequisites
-
Docker & Docker Compose installed on your machine. In my lab setup:-
Docker version:- 27.2.0
Docker Compose version:- Docker Compose version v2.29.2-desktop.2
-
AWS DynamoDB table (for example, vault-ha) set up on AWS.
-
AWS credentials configured locally for Docker containers to access DynamoDB.
Use Case
Though this guide provides you quick and cost-effective setup method to run Vault HA using docker-compose with DynamoDB, however this guide holds values for production use-case too where Vault HA cluster would be running on non-ephemeral or persistent instance or could be on K8s cluster.
Procedure
-
Create a DynamoDB table:-
You need to create a DynamoDB table that Vault will use as the backend. Here is an example of how to do this in the AWS Console or using the AWS CLI:
aws dynamodb create-table \
--table-name vault-ha \
--attribute-definitions AttributeName=lock,AttributeType=S \
--key-schema AttributeName=lock,KeyType=RANGE \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
-
Create a docker-compose file:-
The Docker Compose file will define multiple Vault instances running in HA mode, using DynamoDB as the storage backend.
#version: '3.8'
services:
vault-1:
image: hashicorp/vault:latest
container_name: vault-1
environment:
- AWS_ACCESS_KEY_ID=<redacted>
- AWS_SECRET_ACCESS_KEY=<redacted>
- AWS_SESSION_TOKEN=<redacted>
- VAULT_ADDR=http://127.0.0.1:8200
- VAULT_API_ADDR=http://vault-1:8200
- VAULT_CLUSTER_ADDR=http://vault-1:8201
- VAULT_LOG_LEVEL=debug
- VAULT_UI=true
volumes:
- ./vault-config/vault.hcl:/vault/config/vault.hcl
- vault-1-data:/vault/data
ports:
- "8200:8200"
- "8201:8201"
networks:
- vault-net
command: "server"
#depends_on:
#- vault-2
#- vault-3
restart: unless-stopped
cap_add:
- IPC_LOCK
vault-2:
image: hashicorp/vault:latest
container_name: vault-2
environment:
- AWS_ACCESS_KEY_ID=<redacted>
- AWS_SECRET_ACCESS_KEY=<redacted>
- AWS_SESSION_TOKEN=<redacted>
- VAULT_ADDR=http://127.0.0.1:8200
- VAULT_API_ADDR=http://vault-2:8200
- VAULT_CLUSTER_ADDR=http://vault-2:8201
- VAULT_LOG_LEVEL=debug
- VAULT_UI=true
volumes:
- ./vault-config/vault.hcl:/vault/config/vault.hcl
- vault-2-data:/vault/data
ports:
- "8202:8200"
- "8203:8201"
networks:
- vault-net
command: "server"
#depends_on:
#- vault-1
#- vault-3
restart: unless-stopped
cap_add:
- IPC_LOCK
vault-3:
image: hashicorp/vault:latest
container_name: vault-3
environment:
- AWS_ACCESS_KEY_ID=<redacted>
- AWS_SECRET_ACCESS_KEY=<redacted>
- AWS_SESSION_TOKEN=<redacted>
- VAULT_ADDR=http://127.0.0.1:8200
- VAULT_API_ADDR=http://vault-3:8200
- VAULT_CLUSTER_ADDR=http://vault-3:8201
- VAULT_LOG_LEVEL=debug
- VAULT_UI=true
volumes:
- ./vault-config/vault.hcl:/vault/config/vault.hcl
- vault-3-data:/vault/data
ports:
- "8204:8200"
- "8205:8201"
networks:
- vault-net
command: "server"
#depends_on:
#- vault-1
#- vault-2
restart: unless-stopped
cap_add:
- IPC_LOCK
networks:
vault-net:
driver: bridge
volumes:
vault-1-data:
vault-2-data:
vault-3-data:
-
Vault configuration file (passed reference in docker-compose /vault-config/vault.hcl)
You will also need to create a configuration file vault.hcl for Vault to specify the storage backend and other settings. This will be referenced by each Vault instance in the docker-compose.yml.
# Enable DynamoDB storage backend
storage "dynamodb" {
# DynamoDB table to store Vault's HA state
table = "vault-ha"
ha_enabled = "true"
# Region where your DynamoDB table is located (e.g., us-east-1)
region = "ap-south-1"
# DynamoDB endpoint for local testing (uncomment if using DynamoDB Local)
# endpoint = "http://localhost:8000"
}
# Enable listener on port 8200 (HTTP API)
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_disable = 1 # Disable TLS for testing (set to 0 for production)
}
# Enable the Vault UI (optional)
ui = true
# Configure the Vault HA cluster
api_addr = "http://0.0.0.0:8200"
cluster_addr = "http://0.0.0.0:8201"
# Enable the Vault audit log (optional)
audit {
file_path = "/vault/logs/audit.log"
}
-
Start the Vault cluster
Now you can start your Vault HA cluster using Docker Compose:
docker compose up
This will start three Vault instances running in HA mode, with DynamoDB as the storage backend. The Vault instances will automatically form a cluster using the VAULT_CLUSTER_ADDR and communicate with each other for high availability.
By exec to one of the vault docker container, check the status of vault using below commands.
/ # vault operator members
Host Name API Address Cluster Address Active Node Version Upgrade Version Redundancy Zone Last Echo
--------- ----------- --------------- ----------- ------- --------------- --------------- ---------
c6889579ca79 http://vault-1:8200 https://vault-1:8201 true 1.15.5 n/a n/a n/a
3c3e114f4f84 http://vault-2:8200 https://vault-2:8201 false 1.15.5 n/a n/a 2024-11-09T10:36:14Z
952371a40702 http://vault-3:8200 https://vault-3:8201 false 1.15.5 n/a n/a 2024-11-09T10:36:15Z
/ #
/ #
/ #
/ # vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.15.5
Build Date 2024-01-26T14:53:40Z
Storage Type dynamodb
Cluster Name vault-cluster-89d47aba
Cluster ID 92b64cd2-d68f-073b-07b8-d8258b192a0f
HA Enabled true
HA Cluster https://vault-1:8201
HA Mode active
Active Since 2024-11-09T10:33:52.214880505Z
/ #
Additional Information
-
DynamoDB works differently from the integrated storage backend (raft) where in absence of quorum condition not matching with N/2+1 cluster goes down, however with DynamoDB it kept on running even if let’s say 2 out of 3 vault instance goes down.
/ # vault operator members
Host Name API Address Cluster Address Active Node Version Upgrade Version Redundancy Zone Last Echo
--------- ----------- --------------- ----------- ------- --------------- --------------- ---------
952371a40702 http://vault-3:8200 https://vault-3:8201 true 1.15.5 n/a n/a n/a