Enabling Cross-Namespace Service Communication in Consul with Nomad Service Mesh Jobs – HashiCorp Help Center

Introduction

This article provides guidance on configuring Consul to allow services deployed in different namespaces via Nomad to communicate securely using Consul service mesh and cross-namespace policies.

This article is particularly helpful in scenarios where Consul service mesh communication is required between custom namespaces in VM-based setups. Unlike Kubernetes, Consul does not automatically create cross-namespace policies in VM environments, operators must manually configure them.

Additionally, namespace-level operators in Consul are scoped to their own namespaces by default. Without explicit cross-namespace privileges, they will not be able to view or communicate with services registered in other namespaces.

Expected Outcome

Services deployed using Nomad in separate Consul namespaces (e.g., source and dest) can communicate with each other securely through the Consul Connect service mesh.

Prerequisites

Consul Enterprise v1.14 or later with namespaces and ACLs enabled
Nomad Enterprise v1.8 or later with Consul Connect integration
ACL system bootstrapped and functional in Consul
Admin access to define policies and update namespaces
Registered Consul namespaces: source, dest

Use Case

You want to deploy two services using a single Nomad job file:

count-api service in the dest namespace
count-dashboard in the source namespace

count-dashboard should be able to communicate with count-api over the mesh network.

Procedure

Step 1: Create a Cross-Namespace ACL Policy

Create a policy file named cross-namespace-policy.hcl:

partition "default" {
  namespace_prefix "" {
    node_prefix "" {
      policy = "read"
    }

    service_prefix "" {
      policy = "read"
    }
  }
}

Create this policy in the "default" namespace of Consul via UI or CLI. Below is the CLI command to do that -

consul acl policy create -name cross-namespace-policy -rules=@cross-namespace-policy.hcl

Step 2: Create custom namespaces in the Consul

Create two Consul namespaces named "source" and "dest" via Consul UI or CLI by attaching this policy via UI or CLI. Below is the reference for CLI:

consul namespace update -name=dest -default-policy-name="cross-namespace-policy"
consul namespace update -name=dest -default-policy-name="cross-namespace-policy"

This ensures that any service identity tokens created within these namespaces automatically get access to the necessary rules to communicate across namespaces.

Step 3: Deploy Nomad Job

Use the following sample Nomad job specification to deploy both services and test cross-namespace communication:

job "countdash" {
  group "api" {
    network {
      mode = "bridge"
      port "web" {
        static = 9001
        to     = 9001
      }
    }

    service {
      name = "count-api"
      port = "9001"

      check {
        type     = "http"
        path     = "/health"
        expose   = true
        interval = "3s"
        timeout  = "1s"

        check_restart {
          limit = 0
        }
      }

      connect {
        sidecar_task {
          driver = "docker"
          config {
            image = "envoyproxy/envoy:v1.29.3"
          }
        }

        sidecar_service {
          proxy {}
        }
      }
    }

    consul {
      namespace = "dest"
    }

    task "web" {
      driver = "docker"

      config {
        image          = "hashicorpdev/counter-api:v3"
        auth_soft_fail = true
      }
    }
  }

  group "dashboard" {
    network {
      mode = "bridge"
      port "http" {
        static = 9002
        to     = 9002
      }
    }

    service {
      name = "count-dashboard"
      port = "9002"

      check {
        type     = "http"
        path     = "/health"
        expose   = true
        interval = "3s"
        timeout  = "1s"

        check_restart {
          limit = 0
        }
      }

      connect {
        sidecar_task {
          driver = "docker"
          config {
            image = "envoyproxy/envoy:v1.29.3"
          }
        }

        sidecar_service {
          proxy {
            upstreams {
              destination_name      = "count-api"
              destination_namespace = "dest"
              local_bind_port       = 8080
              local_bind_address    = "127.0.0.1"
            }
          }
        }
      }
    }

    consul {
      namespace = "source"
    }

    task "dashboard" {
      driver = "docker"

      env {
        NOMAD_UPSTREAM_ADDR_count_api = "127.0.0.1:8080"
        COUNTING_SERVICE_URL          = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
      }

      config {
        image          = "hashicorpdev/counter-dashboard:v3"
        auth_soft_fail = true
      }
    }
  }
}

Step 4: Validate

Access the dashboard UI and verify that it successfully queries the count-api from another namespace. Use the Envoy admin interface or Nomad logs to troubleshoot if necessary.

Additional Information