How to implement traffic failover & routing for Consul service hitting external sites via terminating gateway

Avatar
Himanshu Sharma
  • Updated

Introduction

This article explains how to implement traffic failover and routing for HashiCorp Consul services that need to communicate with external sites using a Terminating Gateway. It covers configuration steps, use cases, and best practices to ensure high availability and intelligent routing when accessing external dependencies outside the Consul service mesh.

Expected Outcome

By following this article, users will be able to:

  • Successfully configure a Terminating Gateway in Consul to route service mesh traffic to external endpoints.
  • Implement traffic failover policies to redirect or failover traffic to secondary endpoints when the primary is unavailable.
  • Achieve resilient and controlled outbound communication from Consul services to external systems, ensuring improved reliability and performance.

Prerequisites (if applicable)

  •  A functional consul cluster using guide. Follow the guide to run a Terminating Gateway & register its respective service.
  • External services (targets or sites) are reachable from the gateway nodes.
  • Basic understanding of Consul service-resolver, service-splitter and service-router configurations.

Use Case

  • Enable automatic failover to backup external endpoints if the primary external API or service becomes unreachable.
  • Route outbound service traffic through a centralized Terminating Gateway to enforce egress policies and observability.
  • Ensure only authorized services can access specific external resources via explicit routing and failover definitions.
  • Distribute requests across multiple external services or mirror traffic to a secondary target for canary testing or observability.

Procedure

 
global:
  metrics:
    enabled: true
  name: consul
  enableConsulNamespaces: true
  image: 'hashicorp/consul-enterprise:1.21.0-ent'
  enterpriseLicense:
    secretName: 'consul-ent-license'
    secretKey: 'key'
    enableLicenseAutoload: true
server:
  enabled: true
  replicas: 3
connectInject:
  enabled: true
terminatingGateways:
  enabled: true
 
% kubectl get po -n consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-connect-injector-647db645c-x2p24        1/1     Running   0          2d2h
consul-server-0                                2/2     Running   0          2d2h
consul-server-1                                2/2     Running   0          2d2h
consul-server-2                                2/2     Running   0          2d2h
consul-terminating-gateway-85f7d7bc48-lhphj    1/1     Running   0          47h
consul-webhook-cert-manager-6b46959ffb-9vwrb   1/1     Running   0          2d2h
 
% kubectl exec -it consul-server-0 -n consul -- consul members
Defaulted container "consul" out of: consul, consul-snapshot-agent, locality-init (init)
Node             Address           Status  Type    Build       Protocol  DC   Partition  Segment
consul-server-0  10.244.1.29:8301  alive   server  1.21.0+ent  2         dc1  default    <all>
consul-server-1  10.244.3.17:8301  alive   server  1.21.0+ent  2         dc1  default    <all>
consul-server-2  10.244.2.38:8301  alive   server  1.21.0+ent  2         dc1  default    <all>
 
 
  • Create a sample consul namespace say, sandbox-dev and register 3 instances of external sites using consul catalog API (in the namespace sandbox-dev ) out of which 2 are accessible while the 3rd one (does.not.exist.example.com) isn't valid. 


ext-jsontest- First instance with domaindoes.not.exist.example.com

{
	"ID": "",
	"Node": "ext-sandbox-does.not.exist.example.com",
	"Address": "does.not.exist.example.com",
	"Datacenter": "dc1",
	"TaggedAddresses": null,
	"NodeMeta": {
		"external-node": "true",
		"external-probe": "false",
		"ls_namespace": "sandbox"
	},
	"Service": {
		"ID": "ext-jsontest-98e27464",
		"Service": "ext-jsontest",
		"Tags": [],
		"Address": "does.not.exist.example.com",
		"Weights": {
			"Passing": 1,
			"Warning": 1
		},
		"Meta": {
			"external-source": "terraform",
			"ls_associated_ids": "",
			"ls_namespace": "sandbox",
			"ls_traffic_group": "site-b"
		},
	   	"Port": 80,
		"SocketPath": "",
		"EnableTagOverride": false,
		"Proxy": {
			"Mode": "",
			"MeshGateway": {},
			"Expose": {}
		},
		"Connect": {},
		"Locality": null
	},
	"Partition": "default",
	"Namespace": "sandbox-dev"
}


ext-jsontest- Second instance with domainecho.free.beeceptor.com

{
	"ID": "",
	"Node": "ext-sandbox-echo.free.beeceptor.com",
	"Address": "echo.free.beeceptor.com",
	"Datacenter": "dc1",
	"TaggedAddresses": null,
	"NodeMeta": {
		"external-node": "true",
		"external-probe": "false",
		"ls_namespace": "sandbox"
	},
	"Service": {
	    "ID": "ext-jsontest-6d8cc0cf",
		"Service": "ext-jsontest",
		"Tags": [],
		"Address": "echo.free.beeceptor.com",
		"Weights": {
			"Passing": 1,
			"Warning": 1
		},
		"Meta": {
			"external-source": "terraform",
			"ls_associated_ids": "",
			"ls_namespace": "sandbox",
			"ls_traffic_group": "site-a"
		},
		"Port": 80,
		"SocketPath": "",
		"EnableTagOverride": false,
		"Proxy": {
			"Mode": "",
			"MeshGateway": {},
			"Expose": {}
		},
		"Connect": {},
		"Locality": null
	},
		"Partition": "default",
		"Namespace": "sandbox-dev"
}
 

ext-jsontest- Third instance with domainip.jsontest.com 
{
		"ID": "",
		"Node": "ext-sandbox-ip.jsontest.com",
		"Address": "ip.jsontest.com",
		"Datacenter": "dc1",
		"TaggedAddresses": null,
		"NodeMeta": {
			"external-node": "true",
			"external-probe": "false",
			"ls_namespace": "sandbox"
		},
		"Service": {
		  "ID": "ext-jsontest-d94bc56d",
		  "Service": "ext-jsontest",
		  "Tags": [],
		  "Address": "ip.jsontest.com",
		  "Weights": {
			"Passing": 1,
			"Warning": 1
		  },
		  "Meta": {
			"external-source": "terraform",
			"ls_associated_ids": "",
			"ls_namespace": "sandbox",
			"ls_traffic_group": "default"
		  },
		  "Port": 80,
		  "SocketPath": "",
		  "EnableTagOverride": false,
		  "Proxy": {
			"Mode": "",
			"MeshGateway": {},
			"Expose": {}
		  },
		  "Connect": {},
		  "Locality": null
		},
		"Partition": "default",
		"Namespace": "sandbox-dev"
	}
 
  • We would be creating a few config entries like service-router service-splitter etc for the service ext-jsontest for weighted routing and failover scenario. 

 
service-router --> Define routes and retry parameter for error code

{
	"Kind": "service-router",
	"Name": "ext-jsontest",
	"Routes": [
		{
			"Destination": {
				"Namespace": "sandbox-dev",
				"Partition": "default",
				"NumRetries": 3,
				"RetryOnConnectFailure": true,
				"RetryOn": [
					"gateway-error",
					"reset"
				],
				"RetryOnStatusCodes": [
					503
				]
			}
		}
	],
	"Partition": "default",
	"Namespace": "sandbox-dev"
}

service-splitter --> Define splitting of traffic on the basis of headers or service meta or ServiceSubset. 

{
	"Kind": "service-splitter",
	"Name": "ext-jsontest",
	"Splits": [
		{
			"Weight": 50,
			"Service": "ext-jsontest",
			"ServiceSubset": "site-a",
			"RequestHeaders": {
				"Set": {
					"x-ls-traffic-group": "site-a"
				}
			},
			"ResponseHeaders": {
				"Set": {
					"x-ls-traffic-group": "site-a"
				}
			}
		},
		{
			"Weight": 50,
			"Service": "ext-jsontest",
			"ServiceSubset": "site-b",
			"RequestHeaders": {
				"Set": {
					"x-ls-traffic-group": "site-b"
				}
			},
			"ResponseHeaders": {
				"Set": {
					"x-ls-traffic-group": "site-b"
				}
			}
		}
	],
	"Partition": "default",
	"Namespace": "sandbox-dev"
}

service-resolver --> Define service subsets and failover targets  

{
	"ConnectTimeout": "15s",
	"RequestTimeout": "15s",
	"Kind": "service-resolver",
	"Name": "ext-jsontest",
	"DefaultSubset": "default",
	"Subsets": {
		"default": {
			"Filter": "Service.Meta.ls_traffic_group != \"\""
		},
		"site-a": {
			"Filter": "Service.Meta.ls_traffic_group == \"site-a\"",
			"OnlyPassing": true
		},
		"site-b": {
			"Filter": "Service.Meta.ls_traffic_group == \"site-b\"",
			"OnlyPassing": true
		}
	},
	"Failover": {
		"*": {
			"Targets": [
				{
					"Service": "ext-jsontest",
					"ServiceSubset": "default",
					"Namespace": "sandbox-dev"
				}
			]
		}
	},
	"Partition": "default",
	"Namespace": "sandbox-dev"
}
  • Create a configuration entry for the Terminating Gateway to link the external service named ext-jsontest to it.

Terminating gateway Kubernetes overview | Consul | HashiCorp Developer

apiVersion: consul.hashicorp.com/v1alpha1
kind: TerminatingGateway
metadata:
  name: terminating-gateway
spec:
  services:
   - name: ext-jsontest
     namespace: sandbox-dev
  • Run a application pod to register a downstream service in the consul, which has an upstream connection to the external service ext-jsontest using annotation 'consul.hashicorp.com/connect-service-upstreams': 'ext-jsontest.svc.sandbox-dev.ns:1234'.
apiVersion: v1
kind: Service
metadata:
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      name: static-server
      labels:
        app: static-server
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/connect-service-upstreams': 'ext-jsontest.svc.sandbox-dev.ns:1234'
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http
      serviceAccountName: static-server
Kind      = "proxy-defaults"
Name      = "global"
Config {
  protocol = "http"
  envoy_dns_discovery_type = "STRICT_DNS"
}

 

With the above setup, by checking connectivity from downstream service (ie. static-server) for the failover case  site-b  to the default subset. We don't get any 503 error or no healthy upstream error for failure traffic, instead the traffic request failover to the default subset, which validates that service-resolver works as expected. 

 static-server-6947746cc6-zlk6w  ~  curl -vvs http://localhost:1234
* Host localhost:1234 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:1234...
* connect to ::1 port 1234 from ::1 port 52732 failed: Connection refused
*   Trying 127.0.0.1:1234...
* Connected to localhost (127.0.0.1) port 1234
> GET / HTTP/1.1
> Host: localhost:1234
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< access-control-allow-origin: *
< content-type: application/json
< date: Wed, 04 Jun 2025 11:18:53 GMT
< vary: Accept-Encoding
< x-envoy-upstream-service-time: 303
< server: envoy
< x-ls-traffic-group: site-b
< transfer-encoding: chunked
<
{
  "method": "GET",
  "protocol": "http",
  "host": "echo.free.beeceptor.com",
  "path": "/",
  "ip": "1.6.91.178:55834",
  "headers": {
    "Host": "echo.free.beeceptor.com",
    "User-Agent": "curl/8.7.1",
    "Accept": "*/*",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Forwarded-Client-Cert": "By=spiffe://4b0730ba-47a4-904a-8a19-deb0c38fdb11.consul/ns/sandbox-dev/dc/dc1/svc/ext-jsontest;Hash=a9bf954f52ba94c108de8e54fa18c1f165591d4dc94759492c137a861353af76;Cert=\"-----BEGIN%20CERTIFICATE-----%0AMIICITCCAcegAwIBAgIBEDAKBggqhkjOPQQDAjAwMS4wLAYDVQQDEyVwcmktcTJ0%0AbXZzby5jb25zdWwuY2EuNGIwNzMwYmEuY29uc3VsMB4XDTI1MDYwMzA4MjYxN1oX%0ADTI1MDYwNjA4MjYxN1owADBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABEDV91Dc%0AfuBHH2nSOmx5Xp9S0b1O4W6lfj1vujucPiFmGuaduK4G1bNK3fx5Dr1gakMnX%2F81%0AoeD40HXXeoi6dQ6jggEAMIH9MA4GA1UdDwEB%2FwQEAwIDuDAdBgNVHSUEFjAUBggr%0ABgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH%2FBAIwADApBgNVHQ4EIgQgzROewuY5%0A6N%2FHtvzJ%2FkAnqh7GdHhqYuXKNQ5zcI4QQpUwKwYDVR0jBCQwIoAg1HoBY68GI76f%0A2A3CZPqBThdt3t%2FGWZlD%2FGwqv07IKhUwZgYDVR0RAQH%2FBFwwWoZYc3BpZmZlOi8v%0ANGIwNzMwYmEtNDdhNC05MDRhLThhMTktZGViMGMzOGZkYjExLmNvbnN1bC9ucy9k%0AZWZhdWx0L2RjL2RjMS9zdmMvc3RhdGljLXNlcnZlcjAKBggqhkjOPQQDAgNIADBF%0AAiBd3ztdI0ZIMlSBBdz4t8NdK2A1aWD4HmT52V2OQTqOVgIhAJM8EQm3bmn4aSOf%0AmN%2BdIzBOjkvsPwo8dcoSeqO2dA8a%0A-----END%20CERTIFICATE-----%0A\";Chain=\"-----BEGIN%20CERTIFICATE-----%0AMIICITCCAcegAwIBAgIBEDAKBggqhkjOPQQDAjAwMS4wLAYDVQQDEyVwcmktcTJ0%0AbXZzby5jb25zdWwuY2EuNGIwNzMwYmEuY29uc3VsMB4XDTI1MDYwMzA4MjYxN1oX%0ADTI1MDYwNjA4MjYxN1owADBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABEDV91Dc%0AfuBHH2nSOmx5Xp9S0b1O4W6lfj1vujucPiFmGuaduK4G1bNK3fx5Dr1gakMnX%2F81%0AoeD40HXXeoi6dQ6jggEAMIH9MA4GA1UdDwEB%2FwQEAwIDuDAdBgNVHSUEFjAUBggr%0ABgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH%2FBAIwADApBgNVHQ4EIgQgzROewuY5%0A6N%2FHtvzJ%2FkAnqh7GdHhqYuXKNQ5zcI4QQpUwKwYDVR0jBCQwIoAg1HoBY68GI76f%0A2A3CZPqBThdt3t%2FGWZlD%2FGwqv07IKhUwZgYDVR0RAQH%2FBFwwWoZYc3BpZmZlOi8v%0ANGIwNzMwYmEtNDdhNC05MDRhLThhMTktZGViMGMzOGZkYjExLmNvbnN1bC9ucy9k%0AZWZhdWx0L2RjL2RjMS9zdmMvc3RhdGljLXNlcnZlcjAKBggqhkjOPQQDAgNIADBF%0AAiBd3ztdI0ZIMlSBBdz4t8NdK2A1aWD4HmT52V2OQTqOVgIhAJM8EQm3bmn4aSOf%0AmN%2BdIzBOjkvsPwo8dcoSeqO2dA8a%0A-----END%20CERTIFICATE-----%0A\";Subject=\"\";URI=spiffe://4b0730ba-47a4-904a-8a19-deb0c38fdb11.consul/ns/default/dc/dc1/svc/static-server",
    "X-Ls-Traffic-Group": "site-b",
    "X-Request-Id": "76079e69-b522-4684-aa28-6eca204ce209",
    "Accept-Encoding": "gzip"
  },
  "parsedQueryParams": {}
}* Connection #0 to host localhost left intact

 static-server-6947746cc6-zlk6w  ~  curl -vvs http://localhost:1234
* Host localhost:1234 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:1234...
* connect to ::1 port 1234 from ::1 port 49570 failed: Connection refused
*   Trying 127.0.0.1:1234...
* Connected to localhost (127.0.0.1) port 1234
> GET / HTTP/1.1
> Host: localhost:1234
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< date: Wed, 04 Jun 2025 11:18:58 GMT
< content-type: text/html; charset=UTF-8
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=QvHCkyTry3k6wNeWyofny1IvIkIrNoZXNBigLgAZEdHJr%2FiQqskf%2Bg2aElSAOlOR3e7fyKjnbpFEUp6gZee74Y61xfAi0NHyJyXjPxGumJiZLyHGRbKpztdbFPh%2FwSKkv%2BQ%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: envoy
< cf-ray: 94a708561ae50442-HKG
< alt-svc: h3=":443"; ma=86400
< server-timing: cfL4;desc="?proto=TCP&rtt=186097&min_rtt=161438&rtt_var=45746&sent=38&recv=43&lost=0&retrans=0&sent_bytes=7619&recv_bytes=17840&delivery_rate=17909&cwnd=4&unsent_bytes=0&cid=0000000000000000&ts=0&x=0"
< x-envoy-upstream-service-time: 202
< x-ls-traffic-group: site-b
< transfer-encoding: chunked
<

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>404 Page not found</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Page not found</h1>
<h2>The requested URL was not found on this server.</h2>
<h2></h2>
</body></html>
* Connection #0 to host localhost left intact

 static-server-6947746cc6-zlk6w  ~  curl -vvs http://localhost:1234
* Host localhost:1234 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:1234...
* connect to ::1 port 1234 from ::1 port 49580 failed: Connection refused
*   Trying 127.0.0.1:1234...
* Connected to localhost (127.0.0.1) port 1234
> GET / HTTP/1.1
> Host: localhost:1234
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< date: Wed, 04 Jun 2025 11:19:00 GMT
< content-type: text/html; charset=UTF-8
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=US%2Bzfj8vKtSfx4FjDRtgF2QZ3rt7qGtL6gr35gAILTKxanybw%2BX8%2BHgF8gJbiuxCLmg6ao20LB8SzvxRcdIVnr7eyKosX%2Fq5mFHpVaTOf1K6JyaNeKUBl0RR7aBxznzyfhU%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: envoy
< cf-ray: 94a70862bc53f657-LHR
< alt-svc: h3=":443"; ma=86400
< server-timing: cfL4;desc="?proto=TCP&rtt=191691&min_rtt=181685&rtt_var=53328&sent=29&recv=31&lost=0&retrans=0&sent_bytes=3263&recv_bytes=8920&delivery_rate=15937&cwnd=256&unsent_bytes=0&cid=0000000000000000&ts=0&x=0"
< x-envoy-upstream-service-time: 222
< x-ls-traffic-group: site-a
< transfer-encoding: chunked
<

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>404 Page not found</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Page not found</h1>
<h2>The requested URL was not found on this server.</h2>
<h2></h2>
</body></html>
* Connection #0 to host localhost left intact

 static-server-6947746cc6-zlk6w  ~ 

 

Additional Information

Sidecar proxy configuration reference | Consul | HashiCorp Developer

Envoy proxy configuration reference | Consul | HashiCorp Developer

Proxy defaults configuration entry reference | Consul | HashiCorp Developer

 

Articles in this section