The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

While the official Consul documentation provides a foundation for understanding mesh gateway federation, it may not explicitly outline all the necessary steps for successful implementation. This knowledge base article aims to bridge that gap by providing a comprehensive guide to federating two Consul clusters (dc1 and dc2) using mesh gateways. We'll focus on a common scenario where each datacenter consists of a single server agent running on an EC2 instance.

This guide will walk you through the entire process, including:

• Configuring mesh gateways in both datacenters.
• Establishing TLS connections for secure communication.
• Registering mesh gateways with Consul.
• Verifying successful federation.

By following these detailed instructions, you can confidently establish WAN federation between your Consul clusters and leverage the benefits of a unified service mesh across multiple environments.

Expected Outcome

By the end of this guide, you will have the knowledge and skills to:

• Confidently federate two Consul datacenters (dc1 and dc2) via mesh gateways. Each datacenter will be represented by a single Consul server agent running on an EC2 instance.
• Establish secure communication channels between your datacenters using TLS encryption.
• Successfully configure mesh gateways in both primary and secondary datacenters.
• Verify the functionality of your mesh gateway federation through various tests and checks.

This guide goes beyond the official Consul documentation by providing a clear, step-by-step approach, addressing potential pitfalls, and offering solutions to ensure a smooth and secure WAN federation experience. You'll be able to leverage the full potential of Consul's service mesh across your distributed infrastructure.

Prerequisites

Envoy

Envoy binary must be installed in order to be used by Consul mesh gateway. You can install it via the following link based on your OS flavor.

• Installing Envoy

Note: following the instructions in the above document installs Envoy version 1.18.2.

Consul

This feature is available in Consul versions 1.8.0 and higher

It is better to include the node name in Consul configuration for when the server certificates are created.

• Node Parameters node_name

TLS Configuration for Mesh Gateway Federation

To enable federation through a mesh gateway, TLS is required. Follow these steps to configure TLS certificates:

1. Use the Same CA: Generate server certificates for both datacenters using the same Certificate Authority (CA). This ensures mutual trust between the datacenters.

2. Create Server Certificates with SAN: Generate server certificates for each node in each datacenter. The Subject Alternative Name (SAN) should include the following format:

   <node_name>.server.<this_datacenter>.<domain> (Needed for WAN federation)

    

   • For example: node-dc1.server.dc1.consul

3. Generate Certificates on Each Datacenter: Create the server certificates on each datacenter using the following command:

   sudo consul tls cert create -server -node "node-dc1" -dc "dc1"
   

4. Verify SAN Format: Confirm that the SAN on each certificate includes the correct format. The SAN should include the node name, server name, datacenter name, domain name, and IP address.

   X509v3 Subject Alternative Name:
   DNS:node-dc1.server.dc1.consul, DNS:server.dc1.consul, DNS:localhost, IP Address:127.0.0.1

    

Additional Notes:

• You can find more information about TLS configuration for mesh gateways in the Consul documentation: Enabling WAN Federation Control Plane Traffic
• You can find more information about the consul tls cert create command in the Consul documentation: Consul TLS Cert Create

Procedure

Step 1: Configure Mesh Gateways

To enable WAN federation via mesh gateways, you'll need to configure both your primary and secondary datacenters.

Primary Datacenter (dc1)

Add the following configuration to your Consul configuration file (consul.hcl):

connect {
  enabled = true
  enable_mesh_gateway_wan_federation = true
}

Secondary Datacenter (dc2)

Add the following configuration to your Consul configuration file (consul.hcl):

primary_datacenter = "dc1" 
primary_gateways = ["<primary-mesh-gateway-ip>:<primary-mesh-gateway-port>"]
connect {
  enabled = true
  enable_mesh_gateway_wan_federation = true
}

• Replace <primary-mesh-gateway-ip> and <primary-mesh-gateway-port> with the appropriate values for your primary datacenter's mesh gateway.

Step 2: Register Mesh Gateways

On each datacenter, register the mesh gateway with Consul using the following command:

consul connect envoy -gateway=mesh -expose-servers -register -service "gateway-primary" -address <PRIVATE_IP:PORT> -wan-address "<PUBLIC_IP:PORT>" 

Replace the following placeholders:

• <PRIVATE_IP:PORT>: The private IP address and port of the mesh gateway.
• <PUBLIC_IP:PORT>: The public IP address and port of the mesh gateway.

Important: Ensure that the consul_wan_federation tag is automatically created upon successful launch of the mesh gateway. This tag is crucial for WAN federation to function correctly.

Step 3: Verify Connectivity

After configuring and registering the mesh gateways, verify connectivity between the datacenters:

1. Check WAN Members: Run consul members -wan on a server in each datacenter. This command should list all servers in all datacenters with their local IP addresses and "alive" status.

2. Test API Request: Perform an API request that requires datacenter request forwarding, such as:

   consul catalog services -dc=<OTHER_DATACENTER_NAME>
   

    

   • Replace <OTHER_DATACENTER_NAME> with the name of the other datacenter. This request should succeed, indicating successful communication between the datacenters.

3. Perform KV Test: This test confirms cross-datacenter data replication.

   • From the Secondary Datacenter (dc2):

     consul kv put -datacenter=dc1 test1 test1
     

   • From the Primary Datacenter (dc1):

     consul kv put -datacenter=dc2 test2 test2
     

    

   • Verify that both key-value pairs are accessible from both datacenters.

Additional Information

• Traditional WAN Federation vs. WAN Federation Via Mesh Gateways
• Enabling WAN Federation Control Plane Traffic