Introduction
This guide will walk you through using Consul's external service registration feature and consul-esm to register and monitor the health of a HCP Vault cluster's API endpoint and using Consul DNS to discover and connect to the cluster's Vault API.
Please note using Consul DNS to discover and connect to HCP Vault is not currently recommended for production as it requires API calls to the HCP Vault service name registered in Consul to skip server hostname verification. This is currently required for all API calls to HCP Vault when using this method due to a current limitation around not being able to apply a custom domain/SSL cert to HCP Vault, however, skipping server hostname verification can potentially expose a man in the middle attack.
Prerequisites
- HCP Vault cluster
- Local Consul server agent (this guide uses a local development Consul cluster to demonstrate the concept, but any Consul cluster can be used)
- consul-esm
Procedure
Register HCP Vault to Consul
- Start a local consul agent.
consul agent -dev
- Create a payload file for registering the service. Replace the address and HTTP values below to match the API address of your HCP Vault cluster. Note that the NodeMeta key "external-probe" is set to false because the HCP Vault API hostname does not respond to ping requests, therefore, the node level health check will always fail. Instead, we rely only HTTP health check against the Vault API's health endpoint and we are passing parameters for all nodes (active and standby) in the cluster to return 200s for health checking. You can modify the parameters of the HTTP check URL if needed by referencing this.
cat <<EOF > hcp-vault-external-service.json
{
"Node": "hcp-vault",
"Address": "dev-vault-cluster.vault.11eb2b4c-a26c-a176-bb23-0242ac110005.aws.hashicorp.cloud",
"NodeMeta": {
"external-node": "true",
"external-probe": "false"
},
"Service": {
"ID": "hcp-vault",
"Service": "hcp-vault",
"Port": 8200
},
"Checks": [
{
"Name": "http-check",
"status": "passing",
"Definition": {
"http": "https://dev-vault-cluster.vault.11eb2b4c-a26c-a176-bb23-0242ac110005.aws.hashicorp.cloud:8200/v1/sys/health?standbyok=1&perfstandbyok=1",
"interval": "5s"
}
}
]
}
EOF - Register the HCP Vault address as an external service using the payload file.
curl --request PUT --data @hcp-vault-external-service.json \
http://127.0.0.1:8500/v1/catalog/register - Open a browser to localhost:8500 to see the service registered. It will show a passing HTTP health check status by default.
- You can run consul-esm to have the health check status be updated if the health check fails. By default, consul-esm attempts to connect to localhost:8500 which is where the local development Consul cluster used in this guide is running. If you need to point consul-esm to a different address or modify other configuration options, please refer to consul-esm configuration here.
consul-esm
- The consul UI should show consul-esm registered. To ensure the HCP Vault API is being health checked appropriately, you can intentionally block connectivity (turn off internet connectivity from your machine as an example) to verify and you should see the health check fail.
Discover/Connect to HCP Vault API address using Consul
- Use dig to obtain the IP.
dig @127.0.0.1 -p 8600 hcp-vault.service.consul
; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 8600 hcp-vault.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15803
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hcp-vault.service.consul. IN A
;; ANSWER SECTION:
hcp-vault.service.consul. 0 IN CNAME dev-vault-cluster.vault.11eb2b4c-a26c-a176-bb23-0242ac110005.aws.hashicorp.cloud.
;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Jan 21 10:15:51 CST 2022
;; MSG SIZE rcvd: 147
2. You can also forward DNS queries to Consul to reach the HCP Vault API using the service name, however, at this time since HCP Vault does not support the ability to use a custom domain/SSL cert, you will need to skip TLS verification when making API requests using the service name registered in Consul. Remember - this not currently recommended for production
curl https://hcp-vault.service.consul:8200/v1/sys/health curl: (60) SSL: no alternative certificate subject name matches target host name 'hcp-vault.service.consul' More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. curl https://hcp-vault.service.consul:8200/v1/sys/health -k {"initialized":true,"sealed":false,"standby":false,"performance_standby":false,"replication_performance_mode":"disabled","replication_dr_mode":"disabled","server_time_utc":1642782627,"version":"1.9.2+ent","cluster_name":"vault-cluster-e0dcedf9","cluster_id":"d1ea54b0-592a-9378-c81f-e4a51c3f2666","last_wal":185546,"license":{"state":"autoloaded","expiry_time":"2022-02-26T16:21:06Z","terminated":false}}
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--header "X-Vault-Namespace: admin" \
--request POST \
--data @kvpayload.json \
https://hcp-vault.service.consul:8200/v1/kv/data/my-secret -k
{"request_id":"3ec86a98-3adc-918c-a6eb-88b9fe2d3f90","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2022-01-21T16:52:43.190189348Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}