Consul Wide Area Network (WAN) Federation using Vault as the Certificate Authority (CA)

Avatar
Earl Evans
  • Updated

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Overview

This article will guide you through the configuration steps to set up a Consul WAN Federated environment utilizing Vault as a Certificate Authority, whether it's on Kubernetes or Virtual Machines.

Prerequisites 

Table of Contents

Configure Vault

Configure Vault to use Consul Primary

The first step is to configure all the needed resources within Vault for Consul to properly use it as the CA. 

Consul Storage Engine

  1. Enable the key-value (KV) v2 secrets engine to store Consul secrets in Vault
    vault secrets enable -path=consul kv-v2
  2. If using Consul Enterprise store the enterprise license in Vault
    export consul_license_key="CONSUL-LICENSE-HERE"
vault kv put consul/secret/enterpriselicense key="$consul_license_key"
vault kv get consul/secret/enterpriselicense
  3. Generate a Gossip Key for the Consul cluster and store it in Vault
    export gossip_key="$( uuidgen | tr '[:upper:]' '[:lower:]' )"
vault kv put consul/secret/gossip gossip="$gossip_key"
vault kv get consul/secret/gossip
  4. Generate a Bootstrap Key for the Consul cluster and store it in Vault
    export bootstrap_key="$( uuidgen | tr '[:upper:]' '[:lower:]' )"
vault kv put consul/secret/bootstrap-token token="$bootstrap_key"
vault kv get consul/secret/bootstrap-token
  5. Generate a Replication Key for the Consul cluster and store it in Vault
    export replication_key="$( uuidgen | tr '[:upper:]' '[:lower:]' )"
vault kv put consul/secret/replication-token token="$replication_key"
vault kv get consul/secret/replication-token

 

PKI Storage Engine

Initial PKI

  1. The primary Consul DC will be named "dc1" in this tutorial. If your environment is different, please make sure this is updated. 
    export allowed_pki_domains="dc1.consul, \
consul-server, \
consul-server.consul, \
consul-server.consul.svc"
  2. Enable the PKI engine to be used as the CA root
    vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
  3. Generate root CA cert and save it to the file "consul_ca.crt"
    vault write \
  -field=certificate pki/root/generate/internal \
  common_name="dc1.consul" \
  ttl=87600h | tee consul_ca.crt
  4. Create the PKI role for consul-server
    vault write pki/roles/consul-server \
  allowed_domains="$allowed_pki_domains" \
  allow_subdomains=true \
  allow_bare_domains=true \
  allow_localhost=true \
  generate_lease=true \
  max_ttl="720h"

 

Consul CA Root PKI

  1. Create the PKI path for connect-root 
    vault secrets enable -path connect-root pki
  2. Export the consul controller PKI domains
    export controller_pki_domains="consul-controller-webhook, \
consul-controller-webhook.consul, \
consul-controller-webhook.consul.svc, \
consul-controller-webhook.consul.svc.cluster.local"
  3. Export the connect-injector PKI domains
    export injector_pki_domains="consul-connect-injector, \
consul-connect-injector.consul, \
consul-connect-injector.consul.svc, \
consul-connect-injector.consul.svc.cluster.local"
  4. Create the PKI secrets path for both the Consul controller and connect-inject
    vault secrets enable -path controller pki
vault secrets enable -path connect-inject pki
vault secrets tune -max-lease-ttl 87600h controller
vault secrets tune -max-lease-ttl 87600h connect-inject
  5. Issue the certificates for the Consul controller and connect-inject 
    vault write -field=certificate controller/root/generate/internal \
        common_name="consul-controller-webhook" \
        ttl=87600h
vault write -field=certificate connect-inject/root/generate/internal \
        common_name="consul-connect-injector" \
        ttl=87600h
  6. Write the roles for the Consul controller and connect-inject
    vault write controller/roles/controller-role \
    allowed_domains="$controller_pki_domains" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"
vault write connect-inject/roles/connect-inject-role \
    allowed_domains="$injector_pki_domains" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"
  7. Write the token for Consul Gossip/License/Bootstrap policies
    vault policy write gossip-policy - <<EOF
    path "consul/secret/gossip" {
    capabilities = ["read"]
    }
    EOF

    vault policy write enterpriselicense-policy - <<EOF
    path "consul/secret/enterpriselicense" {
    capabilities = ["read"]
    }
    EOF

    vault policy write bootstrap-token-policy - <<EOF
    path "consul/secret/bootstrap-token" {
    capabilities = ["read"]
    }
    EOF

    vault policy write replication-token-policy - <<EOF
    path "consul/secret/replication-token" {
    capabilities = ["read"]
    }
    EOF
  8. Write the Consul TLS Server and Webhook certificate policies 
    vault policy write controller-tls-ca-policy - <<EOF
    path "/sys/mounts" {
    capabilities = [ "read" ]
    }
    path "controller/issue/controller-role" {
    capabilities = ["create","update"]
    }
    path "controller/cert/ca" {
    capabilities = ["read"]
    }
    EOF

    vault policy write connect-inject-tls-ca-policy - <<EOF
    path "/sys/mounts" {
    capabilities = [ "read" ]
    }
    path "connect-inject/issue/connect-inject-role" {
    capabilities = ["create","update"]
    }
    path "connect-inject/cert/ca" {
    capabilities = ["read"]
    }
    EOF

    vault policy write consul-server - <<EOF
    path "pki/issue/consul-cert-dc1" {
    capabilities = ["create","update"]
    }
    path "auth/token/renew-self" {
    capabilities = [ "update" ]
    }
    path "auth/token/lookup-self" {
    capabilities = ["create","update"]
    }
    EOF

    vault policy write connect-ca-dc1 - <<EOF
    path "/sys/mounts" {
    capabilities = [ "read" ]
    }
    path "/sys/mounts/*" {
    capabilities = ["create","update","read","sudo"]
    }
    path "/sys/mounts/connect_root*" {
    capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "/sys/mounts/connect-intermediate*" {
    capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "/connect-root*" {
    capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "/connect-intermediate*" {
    capabilities = ["create", "read", "update", "delete", "list"]
    }
    EOF

    vault policy write ca-policy - <<EOF
    path "pki/cert/ca" {
    capabilities = ["read"]
    }
    EOF
  9. Fix the associated Consul Secrets Engine policies
    vault write auth/kubernetes/role/consul-server \
  bound_service_account_names="consul-server" \
  bound_service_account_namespaces="consul" \
  policies="consul-server" \
  ttl=24h

vault write auth/kubernetes/role/consul-server-acl-init \
  bound_service_account_names="consul-server-acl-init" \
  bound_service_account_namespaces="consul" \
  policies="consul-server" \
  ttl=24h

vault write auth/kubernetes/role/consul-client \
  bound_service_account_names="consul-client" \
  bound_service_account_namespaces="consul" \
  policies="consul-server"  \
  ttl=24h

vault write auth/kubernetes/role/controller-role \
    bound_service_account_names="consul-controller" \
    bound_service_account_namespaces="consul" \
    policies="consul-server" \
    ttl=1h

vault write auth/kubernetes/role/connect-inject-role \
    bound_service_account_names="consul-connect-injector" \
    bound_service_account_namespaces="consul" \
    policies="connect-ca" \
    ttl=1h

vault write auth/kubernetes/role/consul-ca \
  bound_service_account_names="*" \
  bound_service_account_namespaces="consul" \
  policies="ca-policy" \
  ttl=1h

Configure Vault to use Consul Secondary

  1. Consul TLS Server and Webhook Cert Policies 
    vault policy write consul-cert-dc2 - <<EOF
    path "pki/issue/consul-cert-dc2"
    {
    capabilities = ["create","update"]
    }
    EOF
  2. Consul Secrets Engine Policies ( Need to fix associated policies )
    vault write pki/roles/consul-cert-dc2 \
    allowed_domains="${dc2_allowed_domains}" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

     

Vault Primary DC

  1. Set the Vault Address environment variable
    export VAULT_ADDR=<VAULT-ADDRESS>
  2. Create the Helm values file for the Vault Agent 
     
    cat < primary-values.yaml
server:
  enabled: false
injector:
  enabled: true
  externalVaultAddr: ${VAULT_ADDR}
  authPath: auth/kubernetes-dc1
EOT
  3. Install Vault Agent on the secondary cluster
     
    helm install vault --namespace vault --values primary-values.yaml hashicorp/vault --wait

Configure the Vault Auth Method

  1. Create the manifest file
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-method
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth-method
  namespace: vault
subjects:
- kind: ServiceAccount
  name: vault-auth-method
  namespace: consul
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-method
  namespace: vault
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-method
  namespace: consul
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: "vault-auth-method"
type: kubernetes.io/service-account-token
  2. Apply the manifest in the primary DC
    kubectl apply -f auth-method.yaml
  3. Obtain the appropriate values for the auth method and configure the auth method in Vault
    export K8S_CA_CERT="$(kubectl -n vault get secret/vault-auth-secret --output jsonpath='{.data.ca\.crt}' | base64 --decode)"
export K8S_JWT_TOKEN="$(kubectl -n vault get secret/vault-auth-secret --output jsonpath='{.data.token}' | base64 --decode)"
export KUBE_API_URL=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

    vault auth enable -path=kubernetes-dc1 kubernetes
    vault write auth/kubernetes-dc1/config \ 
        kubernetes_host="${KUBE_API_URL}" \ 
        token_reviewer_jwt="${K8S_JWT_TOKEN}" \ 
        kubernetes_ca_cert="${K8S_CA_CERT}"

 

Vault Secondary DC

  1. Set the Vault Address environment variable 
    export VAULT_ADDR=<VAULT-ADDRESS>
  2. Create the Helm values file for the Vault Agent 
    cat < values.yaml
server:
  enabled: false
injector:
  enabled: true
  externalVaultAddr: ${VAULT_ADDR}
  authPath: auth/kubernetes-dc2
EOT
  3. Install Vault Agent on the secondary cluster
    helm install vault --namespace vault --values values.yaml hashicorp/vault --wait

 

Setup the Vault Auth Method

  1. Create the manifest file
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-method
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth-method
  namespace: vault
subjects:
- kind: ServiceAccount
  name: vault-auth-method
  namespace: consul
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-method
  namespace: vault
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-method
  namespace: consul
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: "vault-auth-method"
type: kubernetes.io/service-account-token
  2. Apply the manifest in the primary DC
    kubectl apply -f auth-method.yaml
  3. Obtain the appropriate values for the auth method and configure the auth method in Vault
    K8S_CA_CERT="$(kubectl -n vault get secret/vault-auth-secret --output jsonpath='{.data.ca\.crt}' | base64 --decode)"
K8S_JWT_TOKEN="$(kubectl -n vault get secret/vault-auth-secret --output jsonpath='{.data.token}' | base64 --decode)"
KUBE_API_URL=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

    vault auth enable -path=kubernetes-dc1 kubernetes
    vault write auth/kubernetes-dc2/config \ 
        kubernetes_host="${KUBE_API_URL}" \ 
        token_reviewer_jwt="${K8S_JWT_TOKEN}" \ 
        kubernetes_ca_cert="${K8S_CA_CERT}"

     

Configure Consul

Using Kubernetes

Consul Primary

  1. Upload the Vault CA as a secret to the cluster (If using SSL/TSL on Vault) 
    kubectl --kubeconfig primary-kubeconfig create secret generic vault-ca-cert \
    --from-file=vault.key=vault.key \
    --from-file=vault.crt=vault.crt \
    --from-file=vault.ca=vault.ca
  2. Create the Consul Helm values file
    cat < consul-primary.yaml
global:
  datacenter: "dc1"
  name: consul
  secretsBackend:
    vault:
      enabled: true
      ca:
        secretKey: vault.ca
        secretName: vault-ca-cert
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
      connectCA:
        address: https://vault.vault.svc.cluster.local:8200
        rootPKIPath: connect_root/
        intermediatePKIPath: dc1/connect_inter/
        authMethodPath: kubernetes-dc1
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: pki/cert/ca
  federation:
    enabled: true
  acls:
    manageSystemACLs: false
    bootstrapToken:
      secretName: 'consul/secret/bootstrap-token'
      secretKey: token
    createReplicationToken: true
    replicationToken:
      secretKey: token
      secretName: 'consul/secret/replication-token'
  gossipEncryption:
    secretName: 'consul/secret/gossip'
    secretKey: gossip
server:
  storage: 50Gi
  resources:
    requests:
      memory: '100Mi'
      cpu: '300m'
    limits:
      memory: '100Mi'
      cpu: '300m'
  replicas: 1
  serverCert:
    secretName: "pki/issue/consul-cert-dc1"
connectInject:
  replicas: 1
  enabled: true
meshGateway:
  enabled: true
  replicas: 1
  service:
    type: LoadBalancer
    port: 8443
ui:
  enabled: true
  service:
    enabled: true
    type: LoadBalancer
EOT
  3. Install Consul on the Primary DC using the supplied values
    helm install consul hashicorp/consul -f consul-primary.yaml --version 0.49.2 --kubeconfig primary-kubeconfig --wait

Consul Secondary

  1. Export the Kubernetes API URL for DC2 and the Mesh Gateway host for the primary DC
    export KUBE_API_URL_DC2=$(kubectl --kubeconfig secondary-kubeconfig config view --minify -o jsonpath='{.clusters[0].cluster.server}')
export MESH_GW_HOST=$(kubectl --kubeconfig primary-kubeconfig get svc consul-mesh-gateway --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
  2. Create the Vault CA cert secret on the secondary DC (If using SSL/TSL on Vault) 
    kubectl --kubeconfig secondary-kubeconfig create secret generic vault-ca-cert \
    --from-file=vault.key=vault.key \
    --from-file=vault.crt=vault.crt \
    --from-file=vault.ca=vault.ca
  3. Generate the Consul Helm values file for the secondary DC
    cat < consul-secondary.yaml
global:
  datacenter: "dc2"
  name: consul
  secretsBackend:
    vault:
      enabled: true
      ca:
        secretKey: vault.ca
        secretName: vault-ca-cert
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
      manageSystemACLsRole: server-acl-init
      connectCA:
        address: ${VAULT_ADDR}
        rootPKIPath: connect_root/
        intermediatePKIPath: dc2/connect_inter/
        authMethodPath: kubernetes-dc2
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: "pki/cert/ca"
  federation:
    enabled: true
    primaryDatacenter: dc1
    k8sAuthMethodHost: ${KUBE_API_URL_DC2}
    primaryGateways:
    - ${MESH_GW_HOST}:8443
  acls:
    manageSystemACLs: false
    replicationToken:
      secretKey: token
      secretName: 'consul/secret/replication-token'
  gossipEncryption:
    secretName: 'consul/secret/gossip'
    secretKey: gossip
server:
  replicas: 1
  storage: 50Gi
  serverCert:
    secretName: "pki/issue/consul-cert-dc2"
connectInject:
  replicas: 1
  enabled: true
controller:
  enabled: true
meshGateway:
  enabled: true
  replicas: 1
  service:
    type: LoadBalancer
    port: 8443
ui:
  enabled: true
  service:
    enabled: true
    type: LoadBalancer
EOT
  4. Create the secondary Consul DC using the values file
    helm install consul hashicorp/consul -f consul-auto-secondary.yaml --version 0.49.2 --kubeconfig secondary-kubeconfig --wait

Using Virtual Machines (VMs)

Consul Primary

  1. Create a Vault Token using the connect root ACL
    vault token create -policy=connect-ca
  2. Add the needed configuration within your primary Consul configuration HCL file
    ## Service mesh CA configuration
connect {
  enabled = true
  ca_provider = "vault"
    ca_config {
        address = "your-vault-address"
        token = "vault-token"
        root_pki_path = "connect_root"
        intermediate_pki_path = "dc1/connect_inter"
        leaf_cert_ttl = "72h"
        rotation_period = "2160h"
        intermediate_cert_ttl = "8760h"
        private_key_type = "rsa"
        private_key_bits = 2048
    }
}

     

Consul Secondary

  1. Add the needed configuration within your primary Consul configuration HCL file
    ## Service mesh CA configuration
connect {
  enabled = true
  ca_provider = "vault"
    ca_config {
        address = "your-vault-address"
        token = "vault-token"
        root_pki_path = "connect_root"
        intermediate_pki_path = "dc2/connect_inter"
        leaf_cert_ttl = "72h"
        rotation_period = "2160h"
        intermediate_cert_ttl = "8760h"
        private_key_type = "rsa"
        private_key_bits = 2048
    }
}

     

Additional Resources

 

Articles in this section

See more

Related articles