Introduction
This tutorial guides you through securing your Consul datacenter on Kubernetes with HashiCorp Vault.
A robust Consul deployment relies on several secrets: gossip encryption keys, TLS certificates for secure server communication, and ACL tokens for granular access control. Manually managing these secrets can be cumbersome and error-prone, especially in dynamic Kubernetes environments.
Vault simplifies this process by providing a centralized and secure platform for storing and distributing secrets. By leveraging Kubernetes service accounts, Vault seamlessly injects authentication tokens into your Consul pods, enabling them to retrieve the necessary secrets without manual intervention.
This approach not only enhances security but also streamlines secret management, eliminating the risk of secret sprawl and simplifying your Consul deployment on Kubernetes.
Expected Outcome
Upon completion of this tutorial, you will be able to:
- Provision a secure Vault instance within your Kubernetes cluster. This includes configuring Vault to use Kubernetes service accounts for authentication.
-
Generate and store Consul's secrets in Vault. This encompasses:
- A gossip encryption key for secure cluster communication.
- TLS certificates for encrypting communication between Consul servers.
- ACL tokens to enforce access control and authorization within Consul.
- Configure Consul to authenticate with Vault. This allows Consul agents to retrieve the necessary secrets directly from Vault.
- Deploy a functional and secure Consul datacenter on Kubernetes. This leverages Vault as the central source of truth for all sensitive information.
By achieving these outcomes, you'll gain practical experience in integrating Vault and Consul within a Kubernetes environment, enabling you to manage secrets effectively and enhance the security of your Consul deployment.
Prerequisites
To get started with this integration, you'll need a Kubernetes (K8s) cluster. There are several ways to set up a multi-node cluster. Feel free to choose the method that best suits your needs and environment.
ubuntu@dc1-master:~$ k get nodes NAME STATUS ROLES AGE VERSION dc1-master Ready control-plane,master 15d v1.24.7+k3s1 dc1-worker2 Ready <none> 15d v1.24.7+k3s1 dc1-worker1 Ready <none> 15d v1.24.7+k3s1
Procedure
Create a Vault Cluster on Kubernetes
- We will create a single server Vault cluster without TLS using its `vault-values.yaml` file below, which creates a single vault server pod and vault-agent-injector.
- If you want to create a cluster with TLS, please refer to the guide.
global: enabled: true tlsDisable: true server: standalone: enabled: true config: | listener "tcp" { address = "[::]:8200" tls_disable = "true" cluster_address = "[::]:8201" } storage "raft" { path = "/vault/data" }
- To get more info on the available helm values configuration options, check the Helm Chart Configuration page.
$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update $ helm install vault -f ./vault-values.yaml hashicorp/vault --version "0.25.0"
- The vault starts uninitialized and in a sealed state.
$ k get po NAME READY STATUS RESTARTS AGE vault-0 0/1 Running 0 32s vault-agent-injector-6549d85b8f-l64fx 1/1 Running 0 32s
- If you want to create a cluster with TLS, please refer to the guide.
- Initialize Vault with one key share and one key threshold.
$ kubectl exec vault-0 -- vault operator init \ -key-shares=1 \ -key-threshold=1 \ -format=json > cluster-keys.json $ kubectl exec vault-0 -- vault operator unseal $(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")
-
Next, verify the
vault-0
pod is running correctly. Ensure its status is1/1
and then execute the following command:kubectl exec -it vault-0 -- vault status
The output should indicate an
HA Mode
ofactive
and aSealed
status offalse
. This confirms that your Vault server is operational and unsealed.$ k get po NAME READY STATUS RESTARTS AGE vault-agent-injector-6549d85b8f-l64fx 1/1 Running 0 11m vault-0 1/1 Running 0 11m
$ vault status
Handling connection for 8200
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.14.0
Build Date 2023-06-19T11:40:23Z
Storage Type raft
Cluster Name vault-cluster-9fe28389
Cluster ID f0594bd1-9be4-69b1-f72e-f91f85851ca6
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Active Since 2023-09-25T16:22:13.442214949Z
Raft Committed Index 2402
Raft Applied Index 2402 - Before deploying the consul helm chart, we need to create all defined secrets and roles inside the Vault cluster.
- Create a KV secret engine to store secrets at path "consul"
$ vault secrets enable -path=consul kv-v2
- Store consul encryption key in Vault
$ vault kv put consul/secret/gossip gossip="$(consul keygen)"
- Create a Vault PKI secrets engine at "pki" path, and tune it to issue certificate with TTL of 10 years.
$ vault secrets enable pki $ vault secrets tune -max-lease-ttl=87600h pki
- Generate the root certificate for Consul CA.
$ vault write -field=certificate pki/root/generate/internal \ common_name="dc1.consul" \ ttl=87600h | tee consul_ca.crt
- Create a role that defines the configuration for the certificates.
$ vault write pki/roles/consul-server \ allowed_domains="dc1.consul,consul-server,consul-server.consul,consul-server.consul.svc" \ allow_subdomains=true \ allow_bare_domains=true \ allow_localhost=true \ generate_lease=true \ max_ttl="720h"
- Enable Vault's PKI secrets engine at the "connect-root" path to be used as root CA for Consul service mesh.
$ vault secrets enable -path connect-root pki
- Create a KV secret engine to store secrets at path "consul"
- Configure Vault's Kubernetes authentication method. This allows your applications to authenticate using Kubernetes Service Account Tokens, granting them access to secrets stored within Vault.
$ vault auth enable kubernetes
-
To ensure compatibility and proper authentication with Vault, we need to create a dedicated service account, secret, and ClusterRoleBinding. These resources will grant Vault the necessary permissions to perform token reviews with Kubernetes and maintain seamless integration.
-
BoundServiceAccountTokenVolume
feature, now enabled by default, alters the default JSON web token (JWT) token mounted within pods.
$ cat <<EOF | kubectl create -f - --- apiVersion: v1 kind: ServiceAccount metadata: name: vault --- apiVersion: v1 kind: Secret metadata: name: vault annotations: kubernetes.io/service-account.name: vault type: kubernetes.io/service-account-token --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: role-tokenreview-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: vault namespace: default EOF
-
- Vault's Kubernetes authentication method allows applications to access secrets using their service account tokens, which Vault verifies against the Kubernetes API server. To enable this, configure Vault with a service account JWT, the Kubernetes CA certificate, and the API server's host URL, establishing secure communication and authorization.
$ TOKEN_REVIEW_JWT=$(kubectl get secret vault -o go-template='{{ .data.token }}' | base64 --decode) $ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode) $ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
- Configure the Vault Kubernetes auth method to use the service account token.
$ vault write auth/kubernetes/config token_reviewer_jwt="$TOKEN_REVIEW_JWT" kubernetes_host="$KUBE_HOST" kubernetes_ca_cert="$KUBE_CA_CERT" disable_local_ca_jwt="true"
- After successful authentication, Vault's policies control which secrets the Consul agent can access. Let's create a policy that grants read access to the "consul" path where your gossip encryption key is stored. This ensures that only authorized Consul agents can retrieve this critical secret.
$ vault policy write gossip-policy - <<EOF path "consul/data/secret/gossip" { capabilities = ["read"] } EOF
- To ensure secure communication, Consul servers require TLS certificates. We'll use Vault's PKI secrets engine to issue certificates for each server (
pki/issue/consul-server
) and provide access to the CA certificate (pki/cert/ca
).
$ vault policy write consul-server - <<EOF path "kv/data/consul-server" { capabilities = ["read"] } path "pki/issue/consul-server" { capabilities = ["read","update"] } path "pki/cert/ca" { capabilities = ["read"] } EOF
- Create a policy named "ca-policy." This policy grants read access to the Consul root CA certificate, allowing Consul agents and services to verify the certificates presented during service-to-service communication.
$ vault policy write ca-policy - <<EOF path "pki/cert/ca" { capabilities = ["read"] } EOF
-
Create a dedicated policy that grants the necessary permissions to create, manage, and issue certificates using Vault's PKI secrets engine. It will allow control over both the root PKI secrets engine, mounted at
connect-root
, and the intermediate engine atconnect-intermediate-dc1
. This ensures that Consul can dynamically generate and distribute certificates for secure service-to-service communication within the mesh.$ vault policy write connect - <<EOF path "/sys/mounts/connect-root" { capabilities = [ "create", "read", "update", "delete", "list" ] } path "/sys/mounts/connect-intermediate-dc1" { capabilities = [ "create", "read", "update", "delete", "list" ] } path "/sys/mounts/connect-intermediate-dc1/tune" { capabilities = [ "update" ] } path "/connect-root/*" { capabilities = [ "create", "read", "update", "delete", "list" ] } path "/connect-intermediate-dc1/*" { capabilities = [ "create", "read", "update", "delete", "list" ] } path "auth/token/renew-self" { capabilities = [ "update" ] } path "auth/token/lookup-self" { capabilities = [ "read" ] } EOF
- To bridge the gap between Kubernetes authentication and Vault's access control, we'll create distinct roles. These roles establish the crucial link between Kubernetes Service Accounts and Vault policies, effectively mapping identities to permissions. This ensures that applications authenticated with specific service accounts are granted the appropriate access levels within Vault, further enhancing security and control.
$ vault write auth/kubernetes/role/consul-server \ bound_service_account_names=consul-server \ bound_service_account_namespaces=consul \ policies="gossip-policy,consul-server,connect" \ ttl=24h $ vault write auth/kubernetes/role/consul-client \ bound_service_account_names=consul-client \ bound_service_account_namespaces=consul \ policies="gossip-policy,ca-policy" \ ttl=24h $ vault write auth/kubernetes/role/consul-ca \ bound_service_account_names="*" \ bound_service_account_namespaces=consul \ policies=ca-policy \ ttl=1h
Create a Consul Cluster
With our Vault configuration complete and our secrets securely stored, we're ready to deploy Consul!
- Example of the Consul Cluster configuration
global: datacenter: "dc1" name: consul domain: consul secretsBackend: vault: # Once enabled, consul will refer to vault to fetch its secrets. enabled: true # K8s Auth Role in Vault that connects the K8s ServiceAccount(consul-server) and its Namespace(consul) with Vault Policies (gossip-policy, consul-server and connect), and returns a token after authentication. consulServerRole: consul-server # K8s Auth Role in Vault that connects the K8s ServiceAccount(consul-client) and its Namespace(consul) with Vault Policies (gossip-policy and ca-policy), and returns a token after authentication. consulClientRole: consul-client # K8s Auth Role in Vault that connects the all K8s ServiceAccount and their Namespace (consul) with Vault Policy (ca-policy), and returns a token after authentication. consulCARole: consul-ca # Below is the configuration of connect CA which creates mTLS certificates for services in K8s and stores it in root and intermediate pki path. connectCA: # Address of connectCA would be vault service, which would be resolved like vault.<namespace>.svc.cluster.local:<port_address> address: http://vault.default:8200 rootPKIPath: connect-root/ intermediatePKIPath: connect-intermediate-dc1/ additionalConfig: "{\"connect\": [{ \"ca_config\": [{ \"namespace\": \"root\"}]}]}" agentAnnotations: | "vault.hashicorp.com/namespace": "root" #As Vault operates in root namespace tls: enabled: true enableAutoEncrypt: true caCert: secretName: "pki/cert/ca" # Path to retrieve CA cert federation: enabled: false createFederationSecret: false acls: manageSystemACLs: false gossipEncryption: secretName: consul/data/secret/gossip #Secret path where `gossip` key is stored secretKey: gossip server: replicas: 1 exposeGossipAndRPCPorts: true serverCert: secretName: "pki/issue/consul-server" #Consul Server to generate TLS certificate connectInject: replicas: 1 enabled: true controller: enabled: false #As controller functionalities is merged within connectInject in helm chart >= 1.0.0 meshGateway: enabled: false replicas: 1 ingressGateways: replicas: 1 enabled: true gateways: - name: ingress-gateway service: type: LoadBalancer terminatingGateways: replicas: 1 enabled: true gateways: - name: terminating-gateway service: type: LoadBalancer ui: enabled: true service: type: LoadBalancer syncCatalog: enabled: true consulNamespaces: mirroringK8S: true k8sDenyNamespaces: ["kube-system", "kube-public"]
- Deploy the Consul Helm chart.
$ helm install --namespace consul --create-namespace \ --wait \ --values ./consul-values.yaml \ consul hashicorp/consul --version "1.1.4" --wait --debug
-
- Verify the installation completed.
$ kubectl get po -n consul NAME READY STATUS RESTARTS AGE consul-connect-injector-7ddfbd84b8-7n8dd 2/2 Running 0 5d consul-ingress-gateway-644b8896d7-phzc6 2/2 Running 0 5d consul-server-0 2/2 Running 0 3d2h consul-sync-catalog-57487bc878-5dx9j 2/2 Running 0 5d consul-terminating-gateway-6d9c6d5f59-wh68p 2/2 Running 0 5d consul-webhook-cert-manager-6c6d66bdd5-4tvr8 1/1 Running 0 5d
- Verify the installation completed.
-
Test the Kubernetes authentication setup. We'll simulate a Consul server login using the
consul-server
service account's JWT and its corresponding Vault role.$ kubectl describe sa consul-server -n consul Name: consul-server Namespace: consul Labels: app=consul app.kubernetes.io/managed-by=Helm chart=consul-helm component=server heritage=Helm release=consul Annotations: meta.helm.sh/release-name: consul meta.helm.sh/release-namespace: consul Image pull secrets: <none> Mountable secrets: consul-server-token-ljg7h Tokens: consul-server-token-ljg7h Events: <none>
-
First, we need to retrieve the JWT token associated with the
consul-server
service account. We'll store this token in an environment variable calledSERVER_JWT
for easy access during our test$ export SERVER_JWT=$(kubectl get secret $(kubectl get sa consul-server -n consul -o jsonpath='{.secrets[0].name}') -n consul -o jsonpath='{.data.token}' | base64 -d)
- Verify the connection.
- The below output indicated a successful connection.
$ curl --request POST --data '{"jwt": "'$SERVER_JWT'", "role": "consul-server"}' http://127.0.0.1:8200/v1/auth/kubernetes/login | jq . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 1798 100 841 100 957 1144 1301 --:--:-- --:--:-- --:--:-- 2456 { "request_id": "93427bba-93e3-7132-a388-c5da5336ea9d", "lease_id": "", "renewable": false, "lease_duration": 0, "data": null, "wrap_info": null, "warnings": null, "auth": { "client_token": "hvs.CAESIDiQktvzXQbocfQ2KlcnIcP9WVYDNT7vOopZ-oMq2rzDGh4KHGh2cy5BZmkxdlFFeVdLckk5NEd4OU0xUUdCaUE", "accessor": "9NxeUbcHVBf8ll9LOEltOIfb", "policies": [ "connect", "consul-server", "default", "gossip-policy" ], "token_policies": [ "connect", "consul-server", "default", "gossip-policy" ], "metadata": { "role": "consul-server", "service_account_name": "consul-server", "service_account_namespace": "consul", "service_account_secret_name": "consul-server-token-ljg7h", "service_account_uid": "e205be82-9338-47f0-94cc-d1a8fd1452b7" }, "lease_duration": 86400, "renewable": true, "entity_id": "5ff5f94b-9c3e-4fe6-6309-5e44b37575d6", "token_type": "service", "orphan": true, "mfa_requirement": null, "num_uses": 0 } }
-
As you can see, the authentication process generated a
client_token
with a 24-hour lease duration. This token grants access to retrieve the necessary secrets, which we'll then securely store at the/vault/secrets/
path.$ kubectl exec -it consul-server-0 -n consul -- sh ~ $ cd /vault/secrets/ /vault/secrets $ ls gossip.txt serverca.crt servercert.crt servercert.key /vault/secrets $ cd /var/run/secrets/kubernetes.io/serviceaccount/ /run/secrets/kubernetes.io/serviceaccount $ ls ca.crt namespace token /run/secrets/kubernetes.io/serviceaccount $
-
As you can see, the authentication process generated a
- The below output indicated a successful connection.
-
It's worth noting that the Consul agent also utilizes the service account's JWT, CA certificate (ca.crt
), and namespace information for its own authentication processes. These files are typically mounted within the pod at the /run/secrets/kubernetes.io/serviceaccount/
path, providing the agent with the necessary credentials to interact with the Kubernetes API server.