INTRODUCTION:
Many of our customers use cloud auto-join features to simplify their node discovery in Vault cluster and maintain the HA. We would like to share best practices and careful considerations to complete the auto-join configuration successfully.
USE CASE:
The customer wanted to use Cloud auto-join feature discovery and maintenance of their cluster’s vault nodes.
https://learn.hashicorp.com/tutorials/vault/raft-deployment-guide?in=vault/raft#raft-configuration
https://learn.hashicorp.com/tutorials/vault/raft-storage-aws?in=vault/raft#cloud-auto-join
BEST PRACTICES:
- Review your vault architecture to choose the cloud auto-join scheme i.e. HTTP or HTTPS.
- The auto-join feature works on <go discover> library which will search the nodes based on their IP and cloud tags.
- Ensure that all your vault nodes and potential nodes are able to communicate with each other over port 8200 and 8201.
- A validation can be performed by : curl -kv http:// ip:8200, 8201
- To configure cloud auto-join stanza for HTTP, use the configuration as per below:
storage "raft" {
path = "/vault/vault_4"
node_id = "vault_4"
retry_join {
auto_join = "provider=aws addr_type=public_v4 tag_key=auto_join tag_value=raft-test region=us-east-1"
auto_join_scheme = "http"
}
}
- In case of AWS ensure your security groups and NACLs allow traffic amongst the tagged EC2 instances between ports 8200 and 8201.
- To view the logs on vault nodes, please use jorunalctl command and you will find
Vault logs show how vault tried to discover other vault node instances and what IPAddress were discovered.
Sample:
[INFO] core: [DEBUG] discover-aws: Found ip addresses: []"
[INFO] core: [DEBUG] discover-aws: Found 0 reservations"
"Sep 29 16:28:19 vault: 2021-09-29T16:28:19.307Z [INFO] core: [INFO] discover-aws: Filter instances with auto_join=raft-test"
- You can leverage AWSCLI to verify if the tagged instances are getting discovered.
aws ec2 describe-instances --filters "Name=tag:auto_join,Values=raft-test" --region ap-southeast-1 | jq '.Reservations[].Instances[].PublicIpAddress'
This will list the the EC2 instances with the tag mentioned.
- For HTTPS Cloud auto-join you will need to ensure that the range of IP addresses is added to the SAN list of the SSL certificate which is used in the architecture.
- Ensure that DNS resolution is happening properly amongst all the nodes and FQDNs.