Consul WAN Federation Setup Between VM-based Primary Datacenter and Kubernetes Secondary Cluster

Avatar
Abhineet Khanwalkar
  • Updated

Overview

This guide demonstrates how to configure WAN federation between a VM-based Consul primary datacenter and a Kubernetes-based secondary datacenter.

The setup includes:

  • A primary Consul cluster running on VMs

  • A secondary Consul cluster running on Kubernetes

  • Mesh gateways for cross-datacenter communication

This configuration enables secure service discovery and service-to-service communication across datacenters.

Architecture Overview

Primary Datacenter (VM)

  • Consul Server running on VM

  • Mesh Gateway running on VM

  • Acts as Primary Datacenter

Secondary Datacenter (Kubernetes)

  • Consul Servers deployed via Helm

  • Mesh Gateway deployed via Consul Helm chart

  • Federation established with the primary cluster

Prerequisites

  • Connectivity between the VM Private IP and Kubernetes Node IPs

  • consul binary and envoy installed on the VM.

Primary Cluster Setup (VM)

1. Generate Security Assets

First, establish the trust domain by creating the Certificate Authority (CA), Server Certificates, and the Gossip encryption key.

# Create CA with custom domain as consul-lab.myinfo.com
consul tls ca create -domain consul-lab.myinfo.com

# Create Server Certificate with SAN for the Secondary DC
consul tls cert create -server -dc dce -node primary-dc1-node \
  -domain=consul-lab.myinfo.com \
  -additional-dnsname="server.dc3-k3s.consul-lab.myinfo.com"

# Generate Gossip Key
consul keygen

2. Configure Consul Server

Create the configuration file at consul.hcl. Note that HTTP is disabled in favor of HTTPS (8501) and gRPC TLS (8502).

# -------------------------
# Core server configuration
# -------------------------
server           = true
bootstrap_expect = 1
datacenter       = "dce"
primary_datacenter = "dce"
domain            = "consul-lab.myinfo.com"
node_name         = "primary-dc1-node"

encrypt = "xOq3gO9a3XXXXXXXXXXXXXXMYk="

bind_addr   = "0.0.0.0"
client_addr = "0.0.0.0"
data_dir    = "/opt/data"
license_path = "/etc/consul.d/license.hclic"


leave_on_terminate = true
enable_debug       = false

# -------------------------
# ACL configuration
# -------------------------
acl {
  enabled                   = true
  default_policy            = "deny"
  down_policy               = "extend-cache"
  enable_token_persistence  = true
  enable_token_replication  = true
  tokens {
    agent = "d20affde-XXXXX-9ea9a7ab4209"
    replication = "d20affde-XXXXX-9ea9a7ab4209"
    default = "d20affde-XXXXX-9ea9a7ab4209"
  }
}

# -------------------------
# Connect / Service Mesh
# -------------------------
connect {
  enabled = true
  enable_mesh_gateway_wan_federation = true
}


enable_central_service_config = true

# -------------------------
# Limits
# -------------------------
limits {
  request_limits {
    mode       = "disabled"
    read_rate  = -1
    write_rate = -1
  }
}

# -------------------------
# Autopilot
# -------------------------
autopilot {
  min_quorum                = 1
  disable_upgrade_migration = true
}

# -------------------------
# Ports (merged from all files)
# -------------------------
ports {
  http      = -1
  https     = 8501
  grpc_tls      = 8502
}

# -------------------------
# TLS configuration
# -------------------------
tls {
  internal_rpc {
    verify_incoming        = true
    verify_outgoing = true
    verify_server_hostname = true
  }

grpc {
    verify_incoming = false
  }


  defaults {
    ca_file   = "/etc/consul.d/tls_cust/consul-lab.myinfo.com-agent-ca.pem"
    cert_file = "/etc/consul.d/tls_cust/dce-server-consul-lab.myinfo.com-0.pem"
    key_file  = "/etc/consul.d/tls_cust/dce-server-consul-lab.myinfo.com-0-key.pem"
  }
}

# -------------------------
# UI
# -------------------------
ui_config {
  enabled = true
}

3. Initialize ACLs and Mesh Gateway

Start Consul and configure the anonymous policy to allow basic service discovery across the federation.

# Start Consul
systemctl start consul

# Create anonymous read policy
echo 'partition_prefix "" { namespace_prefix "" { node_prefix "" { policy = "read" } service_prefix "" { policy = "read" } } }' | consul acl policy create -name anonymous -rules -

# Update anonymous token
consul acl token update -id 00000000-0000-0000-0000-000000000002 -policy-name anonymous

# Start the Mesh Gateway
nohup consul connect envoy -gateway=mesh -register -expose-servers \
  -service "gateway-primary" \
  -address "<private-ip-vm>:8443" \
  -wan-address "<public-ip-vm>:8443" \
  -grpc-addr "https://<private-ip-vm>:8502" \
  -grpc-ca-file "/etc/consul.d/tls_cust/consul-lab.myinfo.com-agent-ca.pem" &

4. Create the Configuration entry for proxy-defaults (proxy-defaults.hcl) on the primary cluster

Kind = "proxy-defaults"
Name = "global"
Config {
  protocol = "http"
}
MeshGateway {
  Mode = "local"
}

 

consul config write proxy-defaults.hcl

# Restart Consul
systemctl restart consul

Kubernetes Secondary Cluster Setup

1. Create Federation Secret

Copy the CA certificates and tokens from the VM to the Kubernetes environment.

kubectl create secret generic consul-federation \
  --from-file=caCert=consul-lab.myinfo.com-agent-ca.pem \
  --from-file=caKey=consul-lab.myinfo.com-agent-ca-key.pem \
  --from-literal=replicationToken="d20affde-XXXXX-9ea9a7ab4209" \
  --from-literal=gossipEncryptionKey="xOq3gO9a3XXXXXXXXXXXXXXMYk="

2. Helm Configuration (values.yaml)

Point the secondary datacenter to the VM's Mesh Gateway IP for initial discovery.

global:
  name: consul
  datacenter: dc3-k3s
  domain: consul-lab.myinfo.com
  image: hashicorp/consul-enterprise:1.21.4-ent
  enterpriseLicense:
    secretName: consul-ent-license
    secretKey: key
  tls:
    enabled: true
    caCert:
      secretName: consul-federation
      secretKey: caCert
    caKey:
      secretName: consul-federation
      secretKey: caKey

  # Delete this acls section if ACLs are disabled.
  acls:
    manageSystemACLs: true
    replicationToken:
      secretName: consul-federation
      secretKey: replicationToken

  federation:
    enabled: true
    k8sAuthMethodHost: https://DB4869DXXXXXXXXXXXX829DD.gr7.us-west-2.eks.amazonaws.com
    primaryDatacenter: dce
    primaryGateways: ["<private-ip-vm>:8443"]

  gossipEncryption:
    secretName: consul-federation
    secretKey: gossipEncryptionKey

connectInject:
  enabled: true
meshGateway:
  enabled: true
server:
  extraConfig: |
    {
      "primary_gateways": ["<private-ip-vm>"]
    }
  enabled: true
  replicas: 1

 

helm install consul hashicorp/consul --values-values.yaml --debug

Validation

Validation Results from Secondary DC:-

Check WAN Members / ACL Replication Status / Cross-DC Catalog Discovery

$ consul members -wan
Node                     Address          Status  Type    Build       Protocol  DC       Partition  Segment
consul-server-0.dc3-k3s  10.0.6.206:8302  alive   server  1.21.4+ent  2         dc3-k3s  default    <all>
primary-dc1-node.dce     10.0.5.32:8302   alive   server  1.21.4+ent  2         dce      default    <all>

$ curl -k https://127.0.0.1:8501/v1/acl/replication?pretty
{
    "Enabled": true,
    "Running": true,
    "SourceDatacenter": "dce",
    "ReplicationType": "tokens",
    "ReplicatedIndex": 45555,
    "ReplicatedRoleIndex": 45556,
    "ReplicatedTokenIndex": 45629,
    "LastSuccess": "2026-02-27T18:48:46Z",
    "LastError": "2026-02-27T18:47:42Z",
    "LastErrorMessage": "failed to retrieve remote ACL tokens: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
}

$ consul catalog services -datacenter=dc3-k3s
consul
mesh-gateway

$ consul catalog services -datacenter=dce
consul
gateway-primaryExpected Output: Both consul-server-0.dc3-k3s and primary-dc1-node.dce should be alive.

Validation Results from Primary DC:-

# consul members -wan
Node                     Address          Status  Type    Build       Protocol  DC       Partition  Segment
consul-server-0.dc3-k3s  10.0.6.206:8302  alive   server  1.21.4+ent  2         dc3-k3s  default    <all>
primary-dc1-node.dce     10.0.5.32:8302   alive   server  1.21.4+ent  2         dce      default    <all>

# consul members -wan
Node                     Address          Status  Type    Build       Protocol  DC       Partition  Segment
consul-server-0.dc3-k3s  10.0.6.206:8302  alive   server  1.21.4+ent  2         dc3-k3s  default    <all>
primary-dc1-node.dce     10.0.5.32:8302   alive   server  1.21.4+ent  2         dce      default    <all>

# consul catalog services -datacenter=dce
consul
gateway-primary

# consul catalog services -datacenter=dc3-k3s
consul
mesh-gateway

 

References:-

https://developer.hashicorp.com/consul/docs/east-west/wan-federation/k8s-vm#kubernetes-as-the-secondary
https://developer.hashicorp.com/consul/docs/east-west/wan-federation/k8s#verifying-federation
https://developer.hashicorp.com/consul/docs/reference/config-entry/proxy-defaults#examples
https://developer.hashicorp.com/consul/docs/reference/k8s/helm
https://developer.hashicorp.com/consul/docs/secure/acl/token/federation#check-replication
https://developer.hashicorp.com/consul/docs/secure/acl/token/federation#create-the-replication-token 

 

Articles in this section

Related articles