Consul & Vault Integration on K8s using Helm Chart

Avatar
Himanshu Sharma
  • Updated

Introduction

This tutorial guides you through securing your Consul datacenter on Kubernetes with HashiCorp Vault.

A robust Consul deployment relies on several secrets: gossip encryption keys, TLS certificates for secure server communication, and ACL tokens for granular access control. Manually managing these secrets can be cumbersome and error-prone, especially in dynamic Kubernetes environments.

Vault simplifies this process by providing a centralized and secure platform for storing and distributing secrets. By leveraging Kubernetes service accounts, Vault seamlessly injects authentication tokens into your Consul pods, enabling them to retrieve the necessary secrets without manual intervention.

This approach not only enhances security but also streamlines secret management, eliminating the risk of secret sprawl and simplifying your Consul deployment on Kubernetes.

Expected Outcome

Upon completion of this tutorial, you will be able to:

  • Provision a secure Vault instance within your Kubernetes cluster. This includes configuring Vault to use Kubernetes service accounts for authentication.
  • Generate and store Consul's secrets in Vault. This encompasses:
    • A gossip encryption key for secure cluster communication.
    • TLS certificates for encrypting communication between Consul servers.
    • ACL tokens to enforce access control and authorization within Consul.
  • Configure Consul to authenticate with Vault. This allows Consul agents to retrieve the necessary secrets directly from Vault.
  • Deploy a functional and secure Consul datacenter on Kubernetes. This leverages Vault as the central source of truth for all sensitive information.

By achieving these outcomes, you'll gain practical experience in integrating Vault and Consul within a Kubernetes environment, enabling you to manage secrets effectively and enhance the security of your Consul deployment.

Prerequisites

To get started with this integration, you'll need a Kubernetes (K8s) cluster. There are several ways to set up a multi-node cluster. Feel free to choose the method that best suits your needs and environment.

  • Create a kind cluster setup
  • Create an EKS cluster
  • Create a K3s cluster setup
This tutorial utilizes a K3s cluster deployed on Multipass, consisting of one master node and two worker nodes.
ubuntu@dc1-master:~$ k get nodes
NAME STATUS ROLES AGE VERSION
dc1-master Ready control-plane,master 15d v1.24.7+k3s1
dc1-worker2 Ready <none> 15d v1.24.7+k3s1
dc1-worker1 Ready <none> 15d v1.24.7+k3s1

Procedure

Create a Vault Cluster on Kubernetes

  1. We will create a single server Vault cluster without TLS using its `vault-values.yaml` file below, which creates a single vault server pod and vault-agent-injector.
    • If you want to create a cluster with TLS, please refer to the guide.
      global:
  enabled: true
  tlsDisable: true

server:
  standalone:
    enabled: true
    config: |
      listener "tcp" {
        address = "[::]:8200"
        tls_disable = "true"
        cluster_address = "[::]:8201"
      }

      storage "raft" {
        path = "/vault/data"
      }
    • To get more info on the available helm values configuration options, check the Helm Chart Configuration page.
      $ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update
$ helm install vault -f ./vault-values.yaml hashicorp/vault --version "0.25.0"
    • The vault starts uninitialized and in a sealed state. 
      $ k get po
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          32s
vault-agent-injector-6549d85b8f-l64fx   1/1     Running   0          32s
  2. Initialize Vault with one key share and one key threshold.
    $ kubectl exec vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > cluster-keys.json
$ kubectl exec vault-0 -- vault operator unseal $(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")

  3. Next, verify the vault-0 pod is running correctly. Ensure its status is 1/1 and then execute the following command:

    kubectl exec -it vault-0 -- vault status

    The output should indicate an HA Mode of active and a Sealed status of false. This confirms that your Vault server is operational and unsealed.

    $ k get po
NAME                                    READY   STATUS    RESTARTS   AGE
vault-agent-injector-6549d85b8f-l64fx   1/1     Running   0          11m
vault-0                                 1/1     Running   0          11m

    $ vault status
    Handling connection for 8200
    Key Value
    --- -----
    Seal Type shamir
    Initialized true
    Sealed false
    Total Shares 1
    Threshold 1
    Version 1.14.0
    Build Date 2023-06-19T11:40:23Z
    Storage Type raft
    Cluster Name vault-cluster-9fe28389
    Cluster ID f0594bd1-9be4-69b1-f72e-f91f85851ca6
    HA Enabled true
    HA Cluster https://vault-0.vault-internal:8201
    HA Mode active
    Active Since 2023-09-25T16:22:13.442214949Z
    Raft Committed Index 2402
    Raft Applied Index 2402
  4. Before deploying the consul helm chart, we need to create all defined secrets and roles inside the Vault cluster.
    1. Create a KV secret engine to store secrets at path "consul"
      $ vault secrets enable -path=consul kv-v2
    2. Store consul encryption key in Vault
      $ vault kv put consul/secret/gossip gossip="$(consul keygen)"
    3. Create a Vault PKI secrets engine at "pki" path, and tune it to issue certificate with TTL of 10 years.
      $ vault secrets enable pki
$ vault secrets tune -max-lease-ttl=87600h pki
    4. Generate the root certificate for Consul CA.
      $ vault write -field=certificate pki/root/generate/internal \
  common_name="dc1.consul" \
  ttl=87600h | tee consul_ca.crt
    5. Create a role that defines the configuration for the certificates.
      $ vault write pki/roles/consul-server \ 
   allowed_domains="dc1.consul,consul-server,consul-server.consul,consul-server.consul.svc" \
   allow_subdomains=true \
   allow_bare_domains=true \
   allow_localhost=true \
   generate_lease=true \
   max_ttl="720h"
    6. Enable Vault's PKI secrets engine at the "connect-root" path to be used as root CA for Consul service mesh.
      $ vault secrets enable -path connect-root pki
  5. Configure Vault's Kubernetes authentication method. This allows your applications to authenticate using Kubernetes Service Account Tokens, granting them access to secrets stored within Vault.
    $ vault auth enable kubernetes

  6. To ensure compatibility and proper authentication with Vault, we need to create a dedicated service account, secret, and ClusterRoleBinding. These resources will grant Vault the necessary permissions to perform token reviews with Kubernetes and maintain seamless integration.

    •  

      Kubernetes 1.21 introduced a significant change to service account token handling. The BoundServiceAccountTokenVolume feature, now enabled by default, alters the default JSON web token (JWT) token mounted within pods.
      $ cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault
---
apiVersion: v1
kind: Secret
metadata:
  name: vault
  annotations:
    kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault
    namespace: default
EOF
  7. Vault's Kubernetes authentication method allows applications to access secrets using their service account tokens, which Vault verifies against the Kubernetes API server. To enable this, configure Vault with a service account JWT, the Kubernetes CA certificate, and the API server's host URL, establishing secure communication and authorization. 
    $ TOKEN_REVIEW_JWT=$(kubectl get secret vault -o go-template='{{ .data.token }}' | base64 --decode) 
$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode) 
$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
  8. Configure the Vault Kubernetes auth method to use the service account token.
    $ vault write auth/kubernetes/config token_reviewer_jwt="$TOKEN_REVIEW_JWT" kubernetes_host="$KUBE_HOST" kubernetes_ca_cert="$KUBE_CA_CERT" disable_local_ca_jwt="true"
  9. After successful authentication, Vault's policies control which secrets the Consul agent can access. Let's create a policy that grants read access to the "consul" path where your gossip encryption key is stored. This ensures that only authorized Consul agents can retrieve this critical secret.
    $ vault policy write gossip-policy - <<EOF 
  path "consul/data/secret/gossip" { 
    capabilities = ["read"] 
  } 
EOF
  10. To ensure secure communication, Consul servers require TLS certificates. We'll use Vault's PKI secrets engine to issue certificates for each server (pki/issue/consul-server) and provide access to the CA certificate (pki/cert/ca). 
    $ vault policy write consul-server - <<EOF
  path "kv/data/consul-server" {
   capabilities = ["read"] 
  } 
  path "pki/issue/consul-server" {
   capabilities = ["read","update"] 
  }
  path "pki/cert/ca" {
   capabilities = ["read"]
  } 
EOF
  11. Create a policy named "ca-policy." This policy grants read access to the Consul root CA certificate, allowing Consul agents and services to verify the certificates presented during service-to-service communication. 
    $ vault policy write ca-policy - <<EOF
  path "pki/cert/ca" {
   capabilities = ["read"]
  } 
EOF

  12. Create a dedicated policy that grants the necessary permissions to create, manage, and issue certificates using Vault's PKI secrets engine. It will allow control over both the root PKI secrets engine, mounted at connect-root, and the intermediate engine at connect-intermediate-dc1. This ensures that Consul can dynamically generate and distribute certificates for secure service-to-service communication within the mesh.

    $ vault policy write connect - <<EOF
  path "/sys/mounts/connect-root" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
  } 
  path "/sys/mounts/connect-intermediate-dc1" {
   capabilities = [ "create", "read", "update", "delete", "list" ] 
  } 
  path "/sys/mounts/connect-intermediate-dc1/tune" {
   capabilities = [ "update" ] 
  } 
  path "/connect-root/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ] 
  } 
  path "/connect-intermediate-dc1/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ] 
  } 
  path "auth/token/renew-self" { 
   capabilities = [ "update" ] 
  } 
  path "auth/token/lookup-self" { 
   capabilities = [ "read" ] 
  } 
EOF
  13. To bridge the gap between Kubernetes authentication and Vault's access control, we'll create distinct roles. These roles establish the crucial link between Kubernetes Service Accounts and Vault policies, effectively mapping identities to permissions. This ensures that applications authenticated with specific service accounts are granted the appropriate access levels within Vault, further enhancing security and control.
    $ vault write auth/kubernetes/role/consul-server \
  bound_service_account_names=consul-server \
  bound_service_account_namespaces=consul \
  policies="gossip-policy,consul-server,connect" \
  ttl=24h 

$ vault write auth/kubernetes/role/consul-client \
  bound_service_account_names=consul-client \
  bound_service_account_namespaces=consul \ 
  policies="gossip-policy,ca-policy" \ 
  ttl=24h 
  
$ vault write auth/kubernetes/role/consul-ca \
  bound_service_account_names="*" \ 
  bound_service_account_namespaces=consul \ 
  policies=ca-policy \ 
  ttl=1h

Create a Consul Cluster

With our Vault configuration complete and our secrets securely stored, we're ready to deploy Consul!

  • Example of the Consul Cluster configuration
    global:
  datacenter: "dc1"
  name: consul
  domain: consul
  secretsBackend:
    vault:
      # Once enabled, consul will refer to vault to fetch its secrets. 
      enabled: true 
      # K8s Auth Role in Vault that connects the K8s ServiceAccount(consul-server) and its Namespace(consul) with Vault Policies (gossip-policy, consul-server and connect), and returns a token after authentication.
      consulServerRole: consul-server
      # K8s Auth Role in Vault that connects the K8s ServiceAccount(consul-client) and its  Namespace(consul) with Vault Policies (gossip-policy and ca-policy), and returns a token after authentication.   
      consulClientRole: consul-client
      # K8s Auth Role in Vault that connects the all K8s ServiceAccount and their Namespace (consul) with Vault Policy (ca-policy), and returns a token after authentication.
      consulCARole: consul-ca
      # Below is the configuration of connect CA which creates mTLS certificates for services in K8s and stores it in root and intermediate pki path.
      connectCA:
        # Address of connectCA would be vault service, which would be resolved like vault.<namespace>.svc.cluster.local:<port_address>
        address: http://vault.default:8200
        rootPKIPath: connect-root/
        intermediatePKIPath: connect-intermediate-dc1/
        additionalConfig: "{\"connect\": [{ \"ca_config\": [{ \"namespace\": \"root\"}]}]}"
      agentAnnotations: |
        "vault.hashicorp.com/namespace": "root"    #As Vault operates in root namespace

  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: "pki/cert/ca"   # Path to retrieve CA cert 
  federation:
    enabled: false
    createFederationSecret: false
  acls:
    manageSystemACLs: false
  gossipEncryption:
    secretName: consul/data/secret/gossip  #Secret path where `gossip` key is stored
    secretKey: gossip

server:
  replicas: 1
  exposeGossipAndRPCPorts: true
  serverCert:
    secretName: "pki/issue/consul-server"   #Consul Server to generate TLS certificate

connectInject:
  replicas: 1
  enabled: true

controller:
  enabled: false #As controller functionalities is merged within connectInject in helm chart >= 1.0.0

meshGateway:
  enabled: false
  replicas: 1

ingressGateways:
  replicas: 1
  enabled: true
  gateways:
    - name: ingress-gateway
      service:
        type: LoadBalancer

terminatingGateways:
  replicas: 1
  enabled: true
  gateways:
    - name: terminating-gateway
      service:
        type: LoadBalancer

ui:
  enabled: true
  service:
    type: LoadBalancer

syncCatalog:
  enabled: true
  consulNamespaces:
    mirroringK8S: true
  k8sDenyNamespaces: ["kube-system", "kube-public"]

     

  1. Deploy the Consul Helm chart.
    $ helm install --namespace consul --create-namespace \
               --wait \
               --values ./consul-values.yaml  \
               consul hashicorp/consul --version "1.1.4" --wait --debug

 

    • Verify the installation completed.
      $ kubectl get po -n consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-connect-injector-7ddfbd84b8-7n8dd       2/2     Running   0          5d
consul-ingress-gateway-644b8896d7-phzc6        2/2     Running   0          5d
consul-server-0                                2/2     Running   0          3d2h
consul-sync-catalog-57487bc878-5dx9j           2/2     Running   0          5d
consul-terminating-gateway-6d9c6d5f59-wh68p    2/2     Running   0          5d
consul-webhook-cert-manager-6c6d66bdd5-4tvr8   1/1     Running   0          5d

  2. Test the Kubernetes authentication setup. We'll simulate a Consul server login using the consul-server service account's JWT and its corresponding Vault role.

    $ kubectl describe sa consul-server -n consul
Name:                consul-server
Namespace:           consul
Labels:              app=consul
                     app.kubernetes.io/managed-by=Helm
                     chart=consul-helm
                     component=server
                     heritage=Helm
                     release=consul
Annotations:         meta.helm.sh/release-name: consul
                     meta.helm.sh/release-namespace: consul
Image pull secrets:  <none>
Mountable secrets:   consul-server-token-ljg7h
Tokens:              consul-server-token-ljg7h
Events:              <none>

    1.  First, we need to retrieve the JWT token associated with the consul-server service account. We'll store this token in an environment variable called SERVER_JWT for easy access during our test

      $ export SERVER_JWT=$(kubectl get secret $(kubectl get sa consul-server -n consul -o jsonpath='{.secrets[0].name}') -n consul -o jsonpath='{.data.token}' | base64 -d)
    2. Verify the connection.
      • The below output indicated a successful connection. 
        $ curl --request POST --data '{"jwt": "'$SERVER_JWT'", "role": "consul-server"}' http://127.0.0.1:8200/v1/auth/kubernetes/login | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1798  100   841  100   957   1144   1301 --:--:-- --:--:-- --:--:--  2456
{
  "request_id": "93427bba-93e3-7132-a388-c5da5336ea9d",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "hvs.CAESIDiQktvzXQbocfQ2KlcnIcP9WVYDNT7vOopZ-oMq2rzDGh4KHGh2cy5BZmkxdlFFeVdLckk5NEd4OU0xUUdCaUE",
    "accessor": "9NxeUbcHVBf8ll9LOEltOIfb",
    "policies": [
      "connect",
      "consul-server",
      "default",
      "gossip-policy"
    ],
    "token_policies": [
      "connect",
      "consul-server",
      "default",
      "gossip-policy"
    ],
    "metadata": {
      "role": "consul-server",
      "service_account_name": "consul-server",
      "service_account_namespace": "consul",
      "service_account_secret_name": "consul-server-token-ljg7h",
      "service_account_uid": "e205be82-9338-47f0-94cc-d1a8fd1452b7"
    },
    "lease_duration": 86400,
    "renewable": true,
    "entity_id": "5ff5f94b-9c3e-4fe6-6309-5e44b37575d6",
    "token_type": "service",
    "orphan": true,
    "mfa_requirement": null,
    "num_uses": 0
  }
}
        • As you can see, the authentication process generated a client_token with a 24-hour lease duration. This token grants access to retrieve the necessary secrets, which we'll then securely store at the /vault/secrets/ path. 
          $ kubectl  exec -it consul-server-0 -n consul -- sh
~ $ cd /vault/secrets/
/vault/secrets $ ls
gossip.txt      serverca.crt    servercert.crt  servercert.key
/vault/secrets $ cd /var/run/secrets/kubernetes.io/serviceaccount/
/run/secrets/kubernetes.io/serviceaccount $ ls
ca.crt     namespace  token
/run/secrets/kubernetes.io/serviceaccount $

It's worth noting that the Consul agent also utilizes the service account's JWT, CA certificate (ca.crt), and namespace information for its own authentication processes. These files are typically mounted within the pod at the /run/secrets/kubernetes.io/serviceaccount/ path, providing the agent with the necessary credentials to interact with the Kubernetes API server.

Additional References

 

Articles in this section

See more

Related articles