Overview
This guide demonstrates how to configure WAN federation between a VM-based Consul primary datacenter and a Kubernetes-based secondary datacenter.
The setup includes:
A primary Consul cluster running on VMs
A secondary Consul cluster running on Kubernetes
Mesh gateways for cross-datacenter communication
This configuration enables secure service discovery and service-to-service communication across datacenters.
Architecture Overview
Primary Datacenter (VM)
Consul Server running on VM
Mesh Gateway running on VM
Acts as Primary Datacenter
Secondary Datacenter (Kubernetes)
Consul Servers deployed via Helm
Mesh Gateway deployed via Consul Helm chart
Federation established with the primary cluster
Prerequisites
Connectivity between the VM Private IP and Kubernetes Node IPs
consulbinary andenvoyinstalled on the VM.
Primary Cluster Setup (VM)
1. Generate Security Assets
First, establish the trust domain by creating the Certificate Authority (CA), Server Certificates, and the Gossip encryption key.
# Create CA with custom domain as consul-lab.myinfo.com
consul tls ca create -domain consul-lab.myinfo.com
# Create Server Certificate with SAN for the Secondary DC
consul tls cert create -server -dc dce -node primary-dc1-node \
-domain=consul-lab.myinfo.com \
-additional-dnsname="server.dc3-k3s.consul-lab.myinfo.com"
# Generate Gossip Key
consul keygen2. Configure Consul Server
Create the configuration file at consul.hcl. Note that HTTP is disabled in favor of HTTPS (8501) and gRPC TLS (8502).
# -------------------------
# Core server configuration
# -------------------------
server = true
bootstrap_expect = 1
datacenter = "dce"
primary_datacenter = "dce"
domain = "consul-lab.myinfo.com"
node_name = "primary-dc1-node"
encrypt = "xOq3gO9a3XXXXXXXXXXXXXXMYk="
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
data_dir = "/opt/data"
license_path = "/etc/consul.d/license.hclic"
leave_on_terminate = true
enable_debug = false
# -------------------------
# ACL configuration
# -------------------------
acl {
enabled = true
default_policy = "deny"
down_policy = "extend-cache"
enable_token_persistence = true
enable_token_replication = true
tokens {
agent = "d20affde-XXXXX-9ea9a7ab4209"
replication = "d20affde-XXXXX-9ea9a7ab4209"
default = "d20affde-XXXXX-9ea9a7ab4209"
}
}
# -------------------------
# Connect / Service Mesh
# -------------------------
connect {
enabled = true
enable_mesh_gateway_wan_federation = true
}
enable_central_service_config = true
# -------------------------
# Limits
# -------------------------
limits {
request_limits {
mode = "disabled"
read_rate = -1
write_rate = -1
}
}
# -------------------------
# Autopilot
# -------------------------
autopilot {
min_quorum = 1
disable_upgrade_migration = true
}
# -------------------------
# Ports (merged from all files)
# -------------------------
ports {
http = -1
https = 8501
grpc_tls = 8502
}
# -------------------------
# TLS configuration
# -------------------------
tls {
internal_rpc {
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
}
grpc {
verify_incoming = false
}
defaults {
ca_file = "/etc/consul.d/tls_cust/consul-lab.myinfo.com-agent-ca.pem"
cert_file = "/etc/consul.d/tls_cust/dce-server-consul-lab.myinfo.com-0.pem"
key_file = "/etc/consul.d/tls_cust/dce-server-consul-lab.myinfo.com-0-key.pem"
}
}
# -------------------------
# UI
# -------------------------
ui_config {
enabled = true
}3. Initialize ACLs and Mesh Gateway
Start Consul and configure the anonymous policy to allow basic service discovery across the federation.
# Start Consul
systemctl start consul
# Create anonymous read policy
echo 'partition_prefix "" { namespace_prefix "" { node_prefix "" { policy = "read" } service_prefix "" { policy = "read" } } }' | consul acl policy create -name anonymous -rules -
# Update anonymous token
consul acl token update -id 00000000-0000-0000-0000-000000000002 -policy-name anonymous
# Start the Mesh Gateway
nohup consul connect envoy -gateway=mesh -register -expose-servers \
-service "gateway-primary" \
-address "<private-ip-vm>:8443" \
-wan-address "<public-ip-vm>:8443" \
-grpc-addr "https://<private-ip-vm>:8502" \
-grpc-ca-file "/etc/consul.d/tls_cust/consul-lab.myinfo.com-agent-ca.pem" &4. Create the Configuration entry for proxy-defaults (proxy-defaults.hcl) on the primary cluster
Kind = "proxy-defaults"
Name = "global"
Config {
protocol = "http"
}
MeshGateway {
Mode = "local"
}
consul config write proxy-defaults.hcl
# Restart Consul
systemctl restart consulKubernetes Secondary Cluster Setup
1. Create Federation Secret
Copy the CA certificates and tokens from the VM to the Kubernetes environment.
kubectl create secret generic consul-federation \
--from-file=caCert=consul-lab.myinfo.com-agent-ca.pem \
--from-file=caKey=consul-lab.myinfo.com-agent-ca-key.pem \
--from-literal=replicationToken="d20affde-XXXXX-9ea9a7ab4209" \
--from-literal=gossipEncryptionKey="xOq3gO9a3XXXXXXXXXXXXXXMYk="2. Helm Configuration (values.yaml)
Point the secondary datacenter to the VM's Mesh Gateway IP for initial discovery.
global:
name: consul
datacenter: dc3-k3s
domain: consul-lab.myinfo.com
image: hashicorp/consul-enterprise:1.21.4-ent
enterpriseLicense:
secretName: consul-ent-license
secretKey: key
tls:
enabled: true
caCert:
secretName: consul-federation
secretKey: caCert
caKey:
secretName: consul-federation
secretKey: caKey
# Delete this acls section if ACLs are disabled.
acls:
manageSystemACLs: true
replicationToken:
secretName: consul-federation
secretKey: replicationToken
federation:
enabled: true
k8sAuthMethodHost: https://DB4869DXXXXXXXXXXXX829DD.gr7.us-west-2.eks.amazonaws.com
primaryDatacenter: dce
primaryGateways: ["<private-ip-vm>:8443"]
gossipEncryption:
secretName: consul-federation
secretKey: gossipEncryptionKey
connectInject:
enabled: true
meshGateway:
enabled: true
server:
extraConfig: |
{
"primary_gateways": ["<private-ip-vm>"]
}
enabled: true
replicas: 1
helm install consul hashicorp/consul --values-values.yaml --debugValidation
Validation Results from Secondary DC:-
Check WAN Members / ACL Replication Status / Cross-DC Catalog Discovery
$ consul members -wan
Node Address Status Type Build Protocol DC Partition Segment
consul-server-0.dc3-k3s 10.0.6.206:8302 alive server 1.21.4+ent 2 dc3-k3s default <all>
primary-dc1-node.dce 10.0.5.32:8302 alive server 1.21.4+ent 2 dce default <all>
$ curl -k https://127.0.0.1:8501/v1/acl/replication?pretty
{
"Enabled": true,
"Running": true,
"SourceDatacenter": "dce",
"ReplicationType": "tokens",
"ReplicatedIndex": 45555,
"ReplicatedRoleIndex": 45556,
"ReplicatedTokenIndex": 45629,
"LastSuccess": "2026-02-27T18:48:46Z",
"LastError": "2026-02-27T18:47:42Z",
"LastErrorMessage": "failed to retrieve remote ACL tokens: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
}
$ consul catalog services -datacenter=dc3-k3s
consul
mesh-gateway
$ consul catalog services -datacenter=dce
consul
gateway-primaryExpected Output: Both consul-server-0.dc3-k3s and primary-dc1-node.dce should be alive.Validation Results from Primary DC:-
# consul members -wan Node Address Status Type Build Protocol DC Partition Segment consul-server-0.dc3-k3s 10.0.6.206:8302 alive server 1.21.4+ent 2 dc3-k3s default <all> primary-dc1-node.dce 10.0.5.32:8302 alive server 1.21.4+ent 2 dce default <all> # consul members -wan Node Address Status Type Build Protocol DC Partition Segment consul-server-0.dc3-k3s 10.0.6.206:8302 alive server 1.21.4+ent 2 dc3-k3s default <all> primary-dc1-node.dce 10.0.5.32:8302 alive server 1.21.4+ent 2 dce default <all> # consul catalog services -datacenter=dce consul gateway-primary # consul catalog services -datacenter=dc3-k3s consul mesh-gateway
References:-
https://developer.hashicorp.com/consul/docs/east-west/wan-federation/k8s-vm#kubernetes-as-the-secondary
https://developer.hashicorp.com/consul/docs/east-west/wan-federation/k8s#verifying-federation
https://developer.hashicorp.com/consul/docs/reference/config-entry/proxy-defaults#examples
https://developer.hashicorp.com/consul/docs/reference/k8s/helm
https://developer.hashicorp.com/consul/docs/secure/acl/token/federation#check-replication
https://developer.hashicorp.com/consul/docs/secure/acl/token/federation#create-the-replication-token