Nomad cross node pools communication by leveraging Consul service discovery – HashiCorp Help Center

Introduction

By default, jobs running in different node pools in HashiCorp Nomad do not directly interact with each other. Each node pool in Nomad is typically isolated in terms of job placement, resource allocation, and network segmentation.

To enable communication between jobs in different node pools, user can use Consul for service discovery. If Nomad jobs register themselves in Consul, other jobs can discover and communicate with them using the service discovery mechanism, regardless of node pool boundaries.

This topic explores how Nomad enables cross-nodepool communication between an API Gateway and backend services by leveraging Consul for service discovery.

Expected Outcome

The expected outcome is a robust, scalable communication setup where the API Gateway dynamically discovers and routes traffic to backend services across different node pools using Consul.

Example:

User will register an API GW & one backend service (using job files) across different node-pools. These jobs would be able to interact with each other using consul service discovery mechanism.

Prerequisites (if applicable)

Ensure to have an integrated consul & nomad cluster using this article. Ref. https://support.hashicorp.com/hc/en-us/articles/39668693975443/live_preview/01JQE4A4WNGX7WSBZ9Z246ZNEH
Need to have a basic understanding around Consul API GWs and its related configuration entries.

Ref. API gateways overview | Consul | HashiCorp Developer

Define API gateway routes on virtual machines | Consul | HashiCorp Developer

Also, need to have basic understanding around proxy-defaults configuration entry to define global configuration parameters for the microservices.

Ref. Proxy defaults configuration entry reference | Consul | HashiCorp Developer

Basic understanding around Docker, Networking & some linux utility commands like nsenter.

Procedure

Step 1: Make followings changes to the Consul Nomad cluster

Create 2 AWS EC2 instances, and then follow the below steps.

On the first EC2 instance-

Create two node pools using the following files.

# node_pool_1.hcl
node_pool "pool_1" {
  description = "Node pool for client_1 workloads."
  meta {
    name = "pool_1"
  }
}

root@ip-172-31-1-65:/home/ubuntu# nomad node pool apply node_pool_1.hcl
Successfully applied node pool "pool_1"!

# node_pool_2.hcl
node_pool "pool_2" {
  description = "Node pool for client_2 workloads."
  meta {
    name = "pool_2"
  }
}

root@ip-172-31-1-65:/home/ubuntu# nomad node pool apply node_pool_2.hcl
Successfully applied node pool "pool_2"!

On second EC2 instance-

Update nomad configuration file to include node_pool.

# nomad_server_config.hcl
ui {
  enabled = true
}
client {
  node_pool = "pool_1"
}

Restart the nomad agent.

On Third EC2-

Run consul & nomad like the second EC2 instance, and use node_pool = “pool_2" in its configuration file.

Step 2: Update the backend service and a API GWs to use different node pools

Update the hello-world sample service by adding the node pool pool_2.

Ref. https://developer.hashicorp.com/nomad/tutorials/manage-jobs/jobs-submit#author-a-job-file

job "hello" {
  node_pool = "pool_2"
  datacenters = ["dc1"]
  group "example" {
    network {
      mode = "bridge"
      port "http" {
        static = "5678"
      }
    }
...

# nomad job status
ID              Type     Priority  Status   Submit Date
hello           service  50        running  2025-03-24T10:37:28Z

Update the first API GW job file to run it on the node pool pool_1.

first_ingress.hcl

job "first-ingress" {
  node_pool = "pool_1" 
  group "first-gateway" {
    count = 1
    network {
...

# nomad job run first_ingress.hcl
==> 2025-03-24T11:05:36Z: Monitoring evaluation "401753c7"
    2025-03-24T11:05:36Z: Evaluation triggered by job "first-ingress"
    ....

API GW related configuration entries like http-route and api-gateway would be intact and stay same.

Step 3: Validating the setup to test the API GW connectivity

With above setup & configuration in place, now we can check/test the connectivity between API GW and backend service.

Run docker ps command to list all running containers for API GW & sample service (hello).
Inspect the docker container to check its PID and then use nsenter utility to check the connectivity to API GWs on its listeners port for customised http-route path.

API GW

# docker inspect --format {{.State.Pid}} 2b70b4b01372
1084793
# nsenter -t 1084793 -n curl localhost:19000/listeners
http:0.0.0.0:8088::0.0.0.0:8088
# nsenter -t 1084793 -n curl localhost:8088/hello
hello world

Conclusion

In conclusion, using Consul for service discovery in Nomad helps nodes easily find and communicate with each other. This improves scalability, reduces delays, and ensures that services stay connected even if there are changes, making the system more reliable and efficient.