Connecting VM-based Workloads to Kubernetes Services via WAN-Federated Mesh Gateways

1. Summary

After establishing WAN Federation between a VM-based Primary Datacenter and a Kubernetes-based Secondary Datacenter, it is critical to validate that the Data Plane (Envoy) can successfully route mTLS traffic across the Mesh Gateways. 

You may refer to the KB article given  here  to setup WAN Federation between VM and Kubernetes environment.

This document provides a standardized procedure for testing Service-to-Service communication using a "Static Client" application on a VM and an "Static Server" application on Kubernetes.

2. Architecture Overview

• Source: static-client (VM / Primary DC)

• Destination: static-server (Kubernetes / Secondary DC)

• Intermediary: Consul Mesh Gateways

• Security: mTLS encrypted with Intentions-based Authorization (RBAC)

3. Implementation Steps

Step 1: Define the Source Service (VM)

On the Primary VM, create a service definition file. This configuration explicitly defines an Upstream dependency pointing to the remote Kubernetes datacenter.

File: static-client.hcl

service {
  name = "static-client"
  port = 4321

  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "static-server"
            datacenter       = "dc3-k3s"   # Remote K8s Datacenter
            local_bind_port  = 1234        # Local loopback port for the app
          }
        ]
      }
    }
  }
}

Step 2: Service Registration & Proxy Initialization

Register the service in the local catalog and initialize the Envoy Sidecar.

Technical Note: In this lab environment, we use a custom admin-bind port (19001) for the sidecar to avoid conflicts with the existing Mesh Gateway process running on the standard 19000 port.

# Register the service
consul services register static-client.hcl

# Start Envoy Sidecar in background
nohup consul connect envoy -sidecar-for static-client -admin-bind "127.0.0.1:19001" &

Step 3: Deploy the Target Workload (Kubernetes)

Deploy the static-server in the secondary cluster. The annotation consul.hashicorp.com/connect-inject: "true" ensures the Envoy sidecar is automatically injected.

File: static-server.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: v1
kind: Service
metadata:
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - port: 1234
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: static-server
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: "true"
      labels:
        app: static-server
    spec:
      serviceAccountName: static-server
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world from dc2"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http

kubectl apply -f static-server.yaml

Step 4: Authorize the Traffic (Zero-Trust)

Consul operates on a "Deny-by-Default" principle. You must explicitly authorize static-client to communicate with static-server.

File: intentions.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: static-client-to-static-server
spec:
  destination:
    name: static-server
  sources:
    - name: static-client
      action: allow

kubectl apply -f intentions.yaml

4. Connectivity Validation & Troubleshooting

From the VM, execute a request to the local upstream listener :

curl localhost:1234

Response Analysis Matrix

5. Engineering Best Practices

• Node Separation: For production environments, do not host Mesh Gateways or application workloads on Consul Server nodes. Use dedicated Consul Client Nodes to ensure server resources are reserved for state management.

• Monitoring: Monitor Envoy's clusters endpoint (curl localhost:19001/clusters) to verify the health status of the remote DC upstream.

• Logging: If connectivity fails, check the Envoy logs for the string TLS_error to identify potential certificate mismatches between the primary and secondary CAs.

6. References

https://developer.hashicorp.com/consul/docs/east-west/wan-federation#network-requirements 

https://developer.hashicorp.com/consul/docs/east-west/wan-federation/k8s-vm 

https://support.hashicorp.com/hc/en-us/articles/49647674531603-Consul-WAN-Federation-Setup-Between-VM-based-Primary-Datacenter-and-Kubernetes-Secondary-Cluster