Customise API GW port through nomad job inside the consul & nomad cluster – HashiCorp Help Center

Introduction

When using HashiCorp's Consul and Nomad clusters, customizing the API Gateway port becomes an essential task, especially in dynamic environments where services are frequently deployed or scaled.

The integration of Consul & Nomad clusters enables seamless communication and dynamic configuration, which is crucial for maintaining the health and availability of the services. However, when deploying an API Gateway within such clusters, customizing its port can be a complex challenge due to the dynamic nature of service discovery and networking. If API GW port is not changed, then on the single client node both API GW ports will run on 8443 port [default port], which will lead to the port collision.

This topic explores how to customize the API Gateway port within a Nomad job, specifically when using it alongside Consul in a Nomad and Consul cluster environment.

Expected Outcome

The expected outcome is to customize the API Gateway port dynamically through Nomad jobs, ensuring that each service is appropriately configured without requiring manual intervention or hard-coded port assignments. This allows user to fine-tune the deployment process to meet network requirements or address potential port conflicts.

Example:

User could run multiple API GWs in the consul & nomad cluster with only 1 single client node, by avoiding port-collision. Let’s say user would run multiple API GWs on ports like 1st API GW on port 8443 & 2nd API GW on port 8444.

Prerequisites (if applicable)

Ensure to have an integrated consul & nomad cluster.
Need to have a basic understanding around Consul API GWs and its related configuration entries.

Ref. API gateways overview | Consul | HashiCorp Developer

Define API gateway routes on virtual machines | Consul | HashiCorp Developer

Also, need to have basic understanding around proxy-defaults configuration entry to define global configuration parameters for the microservices.

Ref. Proxy defaults configuration entry reference | Consul | HashiCorp Developer

Basic understanding around Docker, Networking & some linux utility commands like nsenter.

Procedure

Step 1: Create one minimal configuration consul & nomad cluster

Create two AWS EC2 instances, and then follow the below steps.

# On First EC2:-
# consul_config.hcl
connect {
  enabled = true
}
ports {
  grpc = 8502
}
# 1. Run Consul in server mode 
consul agent -server -ui=true -client=172.31.9.119 -bind 172.31.9.119 -bootstrap=true -log-level=trace -config-file=./consul_config.hcl -data-dir=./Consul_Server/
# nomad_server_config.hcl
ui {
  enabled = true
}
# 2. Run Nomad in server mode
nomad agent -server -bootstrap-expect=1 -bind=172.31.9.119 -log-level=TRACE -data-dir=/home/ubuntu/Nomad_Server/ -config=./nomad_server_config.hcl
# On Second EC2:-
# consul_config.hcl
connect {
  enabled = true
}
ports {
  grpc = 8502
}
# 1. Run Consul in client mode 
consul agent -client -ui=true -retry-join=172.31.9.119 -bind 172.31.4.115 -client 172.31.4.115 -log-level=trace -config-file=./consul_config.hcl -data-dir=./Consul-Client/
# nomad_server_config.hcl
ui {
  enabled = true
}
# 2. Run Nomad in client mode.
nomad agent -client -servers=172.31.9.119 -bind=172.31.4.115 -log-level=TRACE -consul-address=172.31.4.115:8500 -data-dir=/home/ubuntu/Nomad-Client/ -config=./nomad_config.hcl

Once, above setup is in place, then you may see the following output.

# consul members
Node            Address           Status  Type    Build       Protocol  DC   Partition  Segment
ip-172-31-1-65  172.31.1.65:8301  alive   server  1.18.2+ent  2         dc1  default    <all>
ip-172-31-8-7   172.31.8.7:8301   alive   client  1.18.2+ent  2         dc1  default    <default>
# nomad server members
Name                   Address      Port  Status  Leader  Raft Version  Build      Datacenter  Region
ip-172-31-1-65.global  172.31.1.65  4648  alive   true    3             1.9.6+ent  dc1         global
# nomad node status
ID        Node Pool  DC   Name           Class   Drain  Eligibility  Status
58ef1628  default    dc1  ip-172-31-8-7  <none>  false  eligible     ready
root@ip-172-31-8-7:/home/ubuntu#

Step 2: Run the backend service and two API GWs on the same node

Run the hello-world sample service in the cluster.

Ref. Create and submit a job to Nomad | Nomad | HashiCorp Developer

job "hello" {
  datacenters = ["dc1"]
  group "example" {
    network {
      mode = "bridge"
      port "http" {
        static = "5678"
      }
    }
    service {
      name = "hello-world"
      port = "5678"
      connect {
        sidecar_service {
          proxy {
            transparent_proxy {}
          }
        }
      }
    }
    task "server" {
      driver = "docker"
      config {
        image = "hashicorp/http-echo"
        ports = ["http"]
        args = [
          "-listen",
          ":5678",
          "-text",
          "hello world",
        ]
      }
    }
  }
}

# nomad job status
ID              Type     Priority  Status   Submit Date
hello           service  50        running  2025-03-24T10:37:28Z

Run the first API GW and its respective configuration entries (like api-gateway & http-route).

first_ingress.hcl

job "first-ingress" {
  group "first-gateway" {
    count = 1
    network {
      mode = "bridge"
      port "http" {
        static = 8088
        to     = 8088
      }
    }
    task "setup" {
      driver = "docker"
      config {
        image = "hashicorp/consul:latest"
        command = "/bin/sh"
        args = [
          "-c",
          "consul connect envoy -gateway api -address '${attr.unique.network.ip-address}:8443' -bind-address addr='0.0.0.0:8088' -register -deregister-after-critical 10s -service first-api-gateway -admin-bind 0.0.0.0:19000 -ignore-envoy-compatibility -bootstrap > ${NOMAD_ALLOC_DIR}/envoy_bootstrap.json"
        ]
      }
      lifecycle {
        hook = "prestart"
        sidecar = false
      }
      env {
        #CONSUL_HTTP_TOKEN = "5605e580-09a3-a032-60ec-b1802854ee8d"
        CONSUL_HTTP_ADDR = "http://${attr.unique.network.ip-address}:8500"
        CONSUL_GRPC_ADDR = "${attr.unique.network.ip-address}:8502"
      }
    }
    task "first-api" {
      driver = "docker"
      config {
        image = "envoyproxy/envoy:v1.33-latest"
        args = [
          "--config-path",
          "${NOMAD_ALLOC_DIR}/envoy_bootstrap.json",
          "--log-level",
          "${meta.connect.log_level}",
          "--concurrency",
          "${meta.connect.proxy_concurrency}",
          "--disable-hot-restart"
        ]
      }
    }
  }
}

# nomad job run first_ingress.hcl
==> 2025-03-24T11:05:36Z: Monitoring evaluation "401753c7"
    2025-03-24T11:05:36Z: Evaluation triggered by job "first-ingress"
    ....

Note:- User has modified & passed the address & bind-address parameters to the consul connect envoy … command.

Ref. Commands: Connect Envoy | Consul | HashiCorp Developer

first_api_gateway_crd.hcl

Kind = "api-gateway"
Name = "first-api-gateway"
Listeners = [
    {
        name = "listener-one"
        port     = 8088
        protocol = "http"
    }
]

# consul config write first_api_gateway_crd.hcl
Config entry written: api-gateway/first-api-gateway

first_api_gateway_http_route_crd.hcl

Kind = "http-route"
Name = "my-http-route-1"
// Rules define how requests will be routed
Rules = [
  {
    Matches = [
      {
        Path = {
          Match = "prefix"
          Value = "/hello"
        }
      }
    ]
    Services = [
      {
        Name = "hello-world"
      }
    ]
  }
]
Parents = [
  {
    Kind        = "api-gateway"
    Name        = "first-api-gateway"
    SectionName = "listener-one"
  }
]

# consul config write first_api_gateway_http_route_crd.hcl
Config entry written: http-route/my-http-route-1

Run the second API GW and its respective configuration entries (like api-gateway & http-route).

second_ingress.hcl

job "second-ingress" {
  group "second-gateway" {
    count = 1
    network {
      mode = "bridge"
      port "http" {
        static = 8089
        to     = 8089
      }
    }
    task "setup" {
      driver = "docker"
      config {
        image = "hashicorp/consul:latest"
        command = "/bin/sh"
        args = [
          "-c",
          "consul connect envoy -gateway api -address '${attr.unique.network.ip-address}:8444' -bind-address addr='0.0.0.0:8089' -register -deregister-after-critical 10s -service second-api-gateway -admin-bind 0.0.0.0:19001 -ignore-envoy-compatibility -bootstrap > ${NOMAD_ALLOC_DIR}/envoy_bootstrap.json"
        ]
      }
      lifecycle {
        hook = "prestart"
        sidecar = false
      }
      env {
        #CONSUL_HTTP_TOKEN = "5605e580-09a3-a032-60ec-b1802854ee8d"
        CONSUL_HTTP_ADDR = "http://${attr.unique.network.ip-address}:8500"
        CONSUL_GRPC_ADDR = "${attr.unique.network.ip-address}:8502"
      }
    }
    task "second-api" {
      driver = "docker"
      config {
        image = "envoyproxy/envoy:v1.33-latest"
        args = [
          "--config-path",
          "${NOMAD_ALLOC_DIR}/envoy_bootstrap.json",
          "--log-level",
          "${meta.connect.log_level}",
          "--concurrency",
          "${meta.connect.proxy_concurrency}",
          "--disable-hot-restart"
        ]
      }
    }
  }
}

# nomad job run second_ingress.hcl
==> 2025-03-24T11:11:46Z: Monitoring evaluation "0500f6b8"
    2025-03-24T11:11:46Z: Evaluation triggered by job "second-ingress"
    ....

Note:- User has modified & passed the address to pick port 8444 instead of default one 8443, bind-address to port 8089. Also, user has modified admin-bind to port 19001.

Ref. Commands: Connect Envoy | Consul | HashiCorp Developer

second_api_gateway_crd.hcl

Kind = "api-gateway"
Name = "second-api-gateway"
Listeners = [
    {
        name = "listener-two"
        port     = 8089
        protocol = "http"
    }
]

# consul config write second_api_gateway_crd.hcl
Config entry written: api-gateway/second-api-gateway

second_api_gateway_http_route_crd.hcl

Kind = "http-route"
Name = "my-http-route-2"
// Rules define how requests will be routed
Rules = [
  {
    Matches = [
      {
        Path = {
          Match = "prefix"
          Value = "/hello"
        }
      }
    ]
    Services = [
      {
        Name = "hello-world"
      }
    ]
  }
]
Parents = [
  {
    Kind        = "api-gateway"
    Name        = "second-api-gateway"
    SectionName = "listener-two"
  }
]

# consul config write second_api_gateway_http_route_crd.hcl
Config entry written: http-route/my-http-route-2

Lastly, configure proxy-defaults configuration entry to set service protocol to http globally.

Kind      = "proxy-defaults"
Name      = "global"
Config {
  protocol = "http"
}

# consul config write proxy-defaults.hcl
Config entry written: proxy-defaults/global

Step 3: Validting the setup to test the API GW connectivity

With above setup & configuration in place, now user can check/test the connectivity to both API GWs.

Run docker ps command to list all running containers for API GWs & sample service (hello).
Inspect the docker container to check its PID and then use nsenter utility to check the connectivity to API GWs on its listeners port for customised http-route path.

First API GW

# docker inspect --format {{.State.Pid}} 9718e942d853
901417
# nsenter -t 901417 -n curl localhost:19000/listeners
http:0.0.0.0:8088::0.0.0.0:8088
# nsenter -t 901417 -n curl localhost:8088/hello
hello world

Second API GW

# docker inspect --format {{.State.Pid}} dbe0b399f8bd
900687
# nsenter -t 900687 -n curl localhost:19001/listeners
http:0.0.0.0:8089::0.0.0.0:8089
# nsenter -t 900687 -n curl localhost:8089/hello
hello world

Conclusion

Customizing the API Gateway port through Nomad jobs inside a Consul and Nomad cluster is an essential strategy for managing dynamic service architectures in modern microservices environments.

By leveraging Nomad's orchestration capabilities and Consul's service discovery, user can dynamically assign ports to API Gateway, eliminating the need for static port assignments that could lead to conflicts or inefficiencies.