Introduction
This article provides step-by-step guidance to configure Nomad Docker task drivers to enable Consul DNS resolution when using bridge networking mode. This allows allocations to resolve DNS names of services inside the Consul Catalog (usually under .consul
domain)
Expected Outcome
After completing these steps:
Docker containers launched by Nomad will be able to resolve
.consul
DNS names (e.g.,my-service.service.consul
).Applications inside these containers will connect seamlessly to services registered with Consul.
Prerequisites
- A Consul-integrated Nomad cluster using Docker Task Driver
Systemd-resolved managing DNS on the host.
Note: This guide specifically targets environments using systemd-resolved for DNS resolution.
- Access to the host's system configuration, including the ability to modify
systemd
drop-in files and restart services. - Nomad job using Docker with bridge networking mode
Note: This KB is written specifically for Docker Task Drivers.
Use Case
By default, systemd-resolved
listens only on 127.0.0.53
, which is not reachable from containers running in Docker’s bridge network (commonly 172.17.0.1
). As a result, DNS queries for .consul
domains inside these containers fail, breaking service discovery for applications that rely on Consul DNS.
This guide helps you:
Expose
systemd-resolved
on the Docker bridge network interface, so that it can accept DNS queries from container workloads.Configure Nomad jobs to point their containers’ DNS resolution directly at this bridge IP, enabling them to resolve
.consul
service names.
Note: This method does not require Consul itself to run as root, which would otherwise be necessary if Consul needed to bind directly to port
53
for DNS.
Procedure
Step 1: Update systemd-resolved to Route .consul
Queries to Consul DNS
In this step, you will configure systemd-resolved
to forward only DNS queries for domains ending in .consul
to Consul’s DNS server running on the host. This ensures that normal DNS queries (e.g., for google.com
or internal corp DNS) continue using the existing system resolver, while *.consul
lookups are directed to Consul.
- Create a drop-in configuration file for
systemd-resolved
:
sudo mkdir -p /etc/systemd/resolved.conf.d
- Add the following to
/etc/systemd/resolved.conf.d/consul.conf
. This pointssystemd-resolved
to use127.0.0.1:8600
(where Consul’s DNS runs) specifically for.consul
queries, and also exposes the listener on the Docker bridge IP so containers can reach it:
[Resolve]
DNS=127.0.0.1:8600
Domains=.consul
DNSStubListenerExtra=172.17.0.1
Cache=no
DNSSEC=no
- Restart systemd-resolved to apply the changes:
sudo systemctl restart systemd-resolved
Step 2: Update the Nomad Job Specification to Use the Host’s Bridge IP for DNS
Update your Nomad job specification to use the host’s Docker bridge IP for DNS explicitly:
job "httpd" {
group "httpd" {
network {
mode = "bridge"
dns {
servers = ["172.17.0.1"]
}
port "http" {}
}
task "httpd" {
driver = "docker"
config {
image = "busybox:1.36"
command = "httpd"
args = ["-f", "-p", "${NOMAD_PORT_http}"]
ports = ["http"]
}
}
}
}
Step 3: Test DNS Resolution Inside the Container
Exec into the alloc and run a test the DNS resolution for .consul
domains:
nomad alloc exec -task httpd < alloc-id > nslookup consul.service.consul
Server: 172.17.0.1
Address: 172.17.0.1:53
Non-authoritative answer:
Non-authoritative answer:
Name: consul.service.consul
Address: 192.168.105.170
You should now get valid responses.