Understanding Consul Terminating Gateway Configurations for L4/L7 Traffic Routing

Avatar
Nathan Mollica
  • Updated

Introduction

Consul Terminating Gateways (TGWs) are essential for enabling communication between services within the Consul service mesh and external destinations. Acting as service mesh proxies, TGWs terminate mTLS connections, enforce service intentions, and forward requests to appropriate external services. This guide explores how to configure TGWs to handle various protocols (L4/L7) and transport layer security (TLS) settings, ensuring seamless traffic routing.

 

Table of Contents

Service Registration Methods

Explicit Catalog Registration

Explicitly registering external services in Consul's catalog allows Consul to manage them more effectively. This includes:

  • Instance awareness: Consul maintains an accurate count of active service instances, enabling better load balancing decisions.
  • Health monitoring: You can optionally configure Consul ESM (External Service Monitor) to perform health checks on external services, ensuring that traffic is directed only to healthy instances.
  • Enhanced reliability: Consul provides features like load balancing, automatic retries, and failover to ensure that your applications can connect to external services reliably, even in the face of failures.

Configuration:

To demonstrate how Terminating Gateways connect to external services, we'll walk through the manual registration process using Consul's /v1/catalog/register endpoint. This highlights the necessary steps for establishing communication between the service mesh and services residing outside of it.

Generate External Service Registration Payload

Filename: external-service.json

{
  "Node": "example-virtual-node-0",
  "Address": "172.x.x.x", 
  "Partition": "default",
  "Service": {
    "ID": "example-destination",
    "Service": "example-destination",
    "Address": "10.43.194.86",
    "TaggedAddresses": {
      "lan": {
        "address": "10.42.0.25",
        "port": 5432
      }
    },
    "Partition": "default",
    "Port": 5432
  },
  "locality": {
    "region": "us-east-2",
    "zone": "us-east-2a"
  }
}

or for consul-k8s-control-plane ≥ 1.5.1

apiVersion: consul.hashicorp.com/v1alpha1 # required
kind: Registration # required
metadata:
  name: example-destination
spec:
address: "172.x.x.x"
node: example-virtual-node-0
partition: "default"
service:
name: example-destination
port: 5432
namespace: default
partition: default
address: "10.43.194.86"
taggedAddresses:
lan:
address: "10.42.0.25"
port: 5432
locality:
region: "us-east-2"
zone: "us-east-2a"

Register External Service in Consul Catalog

curl --silent \
    --header "X-Consul-Token:${CONSUL_HTTP_TOKEN}" \
    --data @external-service.json \
    --request PUT \
    ${CONSUL_HTTP_ADDR}/v1/catalog/register

Catalog Registration Results

Consul Catalog Nodes

consul catalog nodes

Node                    ID       Address       Partition   DC
consul-server-0         44b729de 10.42.0.20    default     dc1
consul-server-1         bb88a69b 10.42.0.21    default     dc1
consul-server-2         f9d4fd48 10.42.0.22    default     dc1
k8s-node-server-0-virtual        172.18.0.4    default     dc1
example-virtual-node-0 172.x.x.x default dc1

Consul Catalog Services

consul catalog services

example-destination
consul
mesh-gateway
terminating-gateway

Upstream Service Configuration

To connect to the externally accessible service via the Terminating Gateway, a local listener must be exposed on the downstream host (i.e., localhost:5432). Depending on the platform (VM or Kubernetes), this is accomplished two ways.

Consul Connect Proxy Upstream

You would configure the upstream service in the Consul proxy configuration using service-specific proxy configuration files:

service {
  name = "downstream-service"
  connect {
    sidecar_service {
      proxy {
        upstreams = [{
          destination_name = "external-destination"
          local_bind_port  = 5432
        }]
      }
    }
  }
}

Consul Kubernetes Explicit Upstream Annotation

The proxy can be configured to expose the upstream service on a statically configured port by using Kubernetes annotations that are configured on the downstream pod.

annotations:
  "consul.hashicorp.com/connect-service-upstreams":"[service-name].svc:[port]"

DNS Resolution 

nslookup external-destination.service.dc1.consul
Server: 10.43.0.10
Address: 10.43.0.10:53


Name: backend-db.service.dc1.consul
Address: 10.43.212.224

 

ServiceDefaults Destinations (Transparent Proxying)

Transparent proxying allows applications to:

      • Access upstream services via their standard hostname or IP.
      • Avoid application code changes for specific port configurations.

ServiceDefaults Destinations

Services are defined as destinations without full registration via the ServiceDefaults configuration entry. These services:

      • Are not load balanced.
      • Do not support features like retries or failover.

Configuration:

Consul ServiceDefaults Configuration Entry:

Kind = "service-defaults"
Name = "example-destination"
Protocol = "tcp"
Destination {
  Addresses = ["example.com", "example.org"]
  Port = 443
}

Consul Kubernetes ServiceDefaults Resource:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: example-destination
spec:
  protocol: tcp
  destination:
    addresses:
      - "example.com"
      - "example.org"
    port: 443

Destinations Registration Results

Consul Catalog Nodes

Note: No catalog node registered with service

consul catalog nodes

Node                    ID       Address    Partition   DC
consul-server-0         44b729de 10.42.0.20 default     dc1
consul-server-1         bb88a69b 10.42.0.21 default     dc1
consul-server-2         f9d4fd48 10.42.0.22 default     dc1
k3d-c1-server-0-virtual          172.18.0.4 default     dc1

Consul Catalog Services

Note: No catalog service registered with service

consul catalog services

example-destination
consul
mesh-gateway
terminating-gateway

Upstream Service Configuration

To connect to the externally accessible service via the Terminating Gateway, the downstream application can reach the external service via either

      • The Consul-assigned hostname of <service>.virtual.consul, or
      • The upstream service’s IP address or hostname[1].

How Envoy Determines Destination Addresses

      • L4 Traffic: Envoy uses the destination IP from the IP header.
      • L7 Traffic: Envoy inspects headers like Host or :authority for routing decisions. If the traffic is encrypted, Envoy cannot inspect these headers.

DNS Resolution 

Consul Virtual DNS Name

nslookup external-service.virtual.consul
Server: 10.43.0.10
Address: 10.43.0.10:53

Name: backend-db.virtual.consul
Address: 240.0.0.9

Name: backend-db.virtual.consul
Address: 240.0.0.9

Host DNS Resolution

nslookup example.com
Server: 10.43.0.10
Address: 10.43.0.10:53

Name: example.com
Address: 23.192.228.84

Non-authoritative answer:
Name: example.com
Address: 2600:1408:ec00:36::1736:7f31

 

Protocol-Specific Traffic Handling

TCP (L4) Traffic

Unencrypted TCP

Envoy forwards packets based on the destination IP from the IP header.

Configuration: ServiceDefaults Destinations

Kind      = "service-defaults"
Name      = "external-tcp"
Protocol  = "tcp"
Destination {
  Addresses = ["192.0.2.10"]
  Port = 1234
}

Explicit TLS

Envoy uses the SNI field in the TLS handshake to identify the destination hostname for routing.

Configuration:

Kind      = "service-defaults"
Name      = "external-tls"
Protocol  = "tcp"
Destination {
  Addresses = ["example.com"]
  Port = 443
}

Opportunistic TLS

Envoy forwards traffic based on the IP address as TLS negotiation happens after the connection is established.

Configuration:

Kind      = "service-defaults"
Name      = "opportunistic-tls"
Protocol  = "tcp"
Destination {
  Addresses = ["192.0.2.30"]
  Port = 5678
}

HTTP, HTTP/2, gRPC (L7) Traffic

For these protocols, Envoy uses the Host or :authority header to route traffic.

Configuration:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: external-http
spec:
  protocol: http
  destination:
    addresses:
      - "example.org"
    port: 80

 

Database Traffic Examples

MySQL

MySQL uses opportunistic TLS. Envoy routes traffic based on destination IPs.

Configuration:

Kind      = "service-defaults"
Name      = "aws-rds-mysql"
Protocol  = "tcp"
Destination {
  Addresses = ["203.0.113.101", "203.0.113.102"]
  Port = 3306
}

PostgreSQL

Similar to MySQL, PostgreSQL requires routing based on destination IPs.

Configuration:

Kind      = "service-defaults"
Name      = "aws-rds-postgresql"
Protocol  = "tcp"
Destination {
  Addresses = ["198.51.100.101", "198.51.100.102"]
  Port = 5432
}

 

Key Considerations

      • DNS Hostnames: Envoy cannot associate a DNS hostname to incoming traffic unless supported by the protocol.
      • Transparent Proxy: Simplifies application configurations by enabling connections using standard hostnames or IPs.
      • Protocol Awareness: Ensure the protocol and encryption settings align with Envoy’s capabilities.

Benefit Summary

Benefits of Explicit Catalog Registration

      • Full Visibility: Provides complete insight into the number of external service instances.

      • Health Monitoring: Supports health checks via Consul ESM to ensure service availability.

      • Load Balancing: Balances traffic across multiple instances of the service.

      • Resiliency Features: Enables retries, failovers, and other traffic management capabilities.

Benefits of Service Default Destinations

      • Ease of Setup: No need for explicit service registration; simplifies external service integration.

      • Direct IP Routing: Avoids DNS limitations by directly routing traffic to specified IPs or hostnames.

      • Simplified Management: Reduces overhead by focusing only on allowlisting required destinations.

For more information, explore the Consul documentation.

Articles in this section

See more

Related articles