Comprehensive Consul DNS Troubleshooting Guide – HashiCorp Help Center

Introduction

Consul's DNS interface is a powerful feature that allows services registered within Consul to be discovered via standard DNS queries. However, issues such as "name not found", timeouts, or incorrect IP addresses for *.consul queries can arise. This knowledge base article provides a comprehensive guide to diagnose and resolve common Consul DNS resolution problems across various operating systems and environments.

Take-Away: Most Consul DNS incidents boil down to one missing link in a three-layer chain:

The Consul agent is not healthy or reachable.
The OS resolver never asks the agent.
A search path rewrites the question into something Consul does not own.

By following the systematic checklist and procedures outlined in this guide, you will be able to locate – and fix – 90% of resolution problems in minutes instead of hours.

Prerequisites

Before proceeding, ensure you have:

Access to the affected system(s): Command-line access with appropriate privileges (e.g., sudo on Linux, administrator on Windows).
Basic understanding of DNS: Familiarity with DNS concepts like A records, SRV records, and stub resolvers.
Consul agent running: Verify the Consul agent is active and accessible on the target host.
Network connectivity: Confirm basic network reachability between the client and the Consul agent.

Overview

Consul DNS resolution fundamentally relies on a three-layer chain:

Consul Agent DNS Listener: The Consul agent listens for DNS queries on UDP/TCP port 8600 (or 53 if re-mapped). It responds to *.consul queries based on its service catalog.
Host's Stub Resolver: The operating system's local DNS resolver (e.g., systemd-resolved, dnsmasq, Unbound, libc stub resolver) is configured to forward *.consul queries to the local Consul agent's DNS listener (typically 127.0.0.1:8600).
Application Query: The application makes a DNS request through the operating system's standard getaddrinfo (or similar) functions, which then interacts with the stub resolver.

The simplified flow looks like this:

application ↔ libc stub (getaddrinfo)
            ↔ 127.0.0.53:53  ← systemd‑resolved
            ↔ 127.0.0.1:8600 ← Consul agent DNS listener
            ↔ Consul servers → Service catalog / upstream recursors

If any link in this chain is broken, DNS resolution for *.consul names will fail. This guide will help you identify which link is the culprit and provide solutions.

Scope: This guide applies to Linux (with and without systemd-resolved), macOS, Windows, containers, and network appliances that forward DNS to a local Consul agent.

Procedures

Follow these procedures systematically to diagnose and resolve Consul DNS issues.

1. Quick "Is It Consul or the OS?" Checklist

Start by performing these checks to quickly pinpoint the source of the problem.

Step	Command	Healthy result
1	`dig @127.0.0.1 -p 8600 consul.service.consul`	Return has `ANSWER SECTION` Example: ;; ANSWER SECTION: consul.service.consul. 10 IN A 10.0.132.196 consul.service.consul. 10 IN A 10.0.84.142 consul.service.consul. 10 IN A 10.0.7.173
2	`resolvectl query consul.service.consul` (Linux) / `host` (macOS) / `Resolve-DnsName` (Windows)	Same IP(s) as step 1 consul.service.consul: 10.0.7.173 10.0.84.142 10.0.132.196 -- Information acquired via protocol DNS in 1.3ms. -- Data is authenticated: no; Data was acquired via local or encrypted transport: no -- Data from: network
3	`consul catalog services \| grep my-service`	Returns matching result with name `my-service`
4	`sudo iptables -L -t nat \| grep 8600`	REDIRECT udp -- anywhere localhost udp dpt:domain redir ports 8600 REDIRECT tcp -- anywhere localhost tcp dpt:domain redir ports 8600
5	`consul monitor`	No ACL “permission denied” / “permission denied for dns request” messages

2. OS-Level Resolver Pitfalls

If Step 2 in the checklist failed, the issue likely lies with how your operating system's DNS resolver is configured to forward *.consul queries.

2.1. `systemd-resolved` (Most Modern Linux)

This is a common source of issues due to its caching and domain-routing behavior.

Best Practice: Create a dedicated drop-in configuration file for Consul. This keeps vendor files intact and simplifies management.
- Create /etc/systemd/resolved.conf.d/consul.conf with the following content:
  [Resolve] DNS=127.0.0.1:8600 Domains=~consul
- The ~consul entry makes it a route-only domain, preventing single-label name rewrites (e.g., preventing vault from becoming vault.consul).
Important Note for systemd v246+: Avoid DNS=127.0.0.1 without a port if your Consul agent still listens on 8600. Without the explicit port, systemd-resolved will send queries to port 53, where Consul is not listening by default.
Older systemd versions (<246): Omit the port (e.g., DNS=127.0.0.1) or use iptables to redirect port 53 to 8600.
Flush Cache: After making changes, always flush the systemd-resolved cache as it caches negative answers:
```
sudo resolvectl flush-caches
sudo systemctl restart systemd-resolved
```
Verify Routing: Check the resolvectl domain output; it should show Global: ~consul or similar.
Multiple Network Links: If your system has multiple network interfaces, ensure the Consul stub resolver configuration applies to the link that "owns" the 127.0.0.1 loopback address.
MulticastDNS Conflicts: If you observe .local collisions or delays, consider disabling MulticastDNS:
```
# In /etc/systemd/resolved.conf or a drop-in file
MulticastDNS=no
```
.local mDNS traffic can sometimes cause query delays.
FAQ: Domains=local ~consul? This line typically does not break Consul. It only appends .local to single-label host queries. Fully-qualified names like vault.service.consul will still be routed correctly via the ~consul stub. You can remove local if you wish to keep single-label names untouched.

2.2. Traditional `/etc/resolv.conf` (No Stub Resolver)

If your system directly uses /etc/resolv.conf without a separate stub resolver like systemd-resolved, add a trailing entry:

# /etc/resolv.conf
nameserver 127.0.0.1
port 8600       # if your resolver supports this non-standard option

Alternatively, configure your Consul agent to listen on port 53 directly.

2.3. `dnsmasq` / `Unbound` / `BIND`

For systems using these DNS forwarders/resolvers, configure them to send *.consul queries to the local Consul agent:

dnsmasq example: Create a file like /etc/dnsmasq.d/10-consul.conf
```
server=/consul/127.0.0.1#8600
```
Refer to the Consul documentation for more comprehensive examples tailored to each resolver.

2.4. macOS

Create a resolver configuration file for the .consul domain:

sudo mkdir -p /etc/resolver
echo -e "nameserver 127.0.0.1\nport 8600" | sudo tee /etc/resolver/consul

2.5. Windows (Enterprise Editions)

Add a Name Resolution Policy Table (NRPT) rule to direct .consul queries:

Add-DnsClientNrptRule -Namespace ".consul" `
                      -NameServers "127.0.0.1,::1" `
                      -Comment "Route Consul DNS locally"

3. Consul-Side Causes

If Step 1 of the checklist (direct dig to Consul agent) failed, the problem is within your Consul agent's operation or configuration.

Symptom	Consul check	Fix
`connection refused` from dig	`netstat -plntu \| grep 8600`	Ensure Consul agent is started and DNS port is configured for port 8600
Random time‑outs	`consul operator raft list-peers`	Remove unhealthy servers; ensure quorum.
Wrong IPs / stale SRV	`consul catalog services`	Verify service is registered in correct datacenter, tags, and address.
NXDOMAIN for service	Check ACLs: `consul acl policy read -name dns-read`	Grant `query_prefix "" { policy = read }`.
Delayed answers	`consul monitor --log-level=tracec`	Monitor for ERRORs pertaining to DNS queries or network misconfiguration, and correct.

Additional Information

End-to-End Diagnostic Workflow

For deeper investigation, follow this comprehensive workflow:

Verify Agent Health: Check the Consul agent's overall health and DNS configuration:
```
consul info
```
Confirm Port Binding: Ensure the Consul agent is correctly listening on the expected port:
```
ss -lupn | grep 8600
```
Query Consul Directly: Re-run Step 1 from the "Quick Checklist" to confirm the agent's direct responsiveness.
```
dig @127.0.0.1 -p 8600 consul.service.consul
```
Inspect OS Routing: Check how your OS handles DNS queries for the .consul domain.
- Linux (systemd-resolved): resolvectl status
- macOS: cat /etc/resolver/consul
- Windows: Get-DnsClientNrptRule
Watch Packets: Use tcpdump to observe if DNS queries are actually reaching the Consul agent's port on the loopback interface:
```
sudo tcpdump -i lo udp port 8600
```
Run a dig command from another terminal while tcpdump is running.
Trace Recursion (Advanced): If the agent is responding but the resolution path is unclear, try tracing the DNS query:
```
dig +trace consul.service.consul @127.0.0.1 -p 8600
```
Flush Caches: Clear any system-level DNS caches that might be holding stale or negative entries.
- Linux (systemd-resolved): resolvectl flush-caches
- Windows: ipconfig /flushdns
- macOS: dscacheutil -flushcache; sudo killall -HUP mDNSResponder
Check Firewalls / SELinux / AppArmor: Confirm that udp/8600 (and tcp/8600 if needed) are not blocked by local firewalls (e.g., ufw, firewalld, iptables) or security enhancements like SELinux or AppArmor.
Overlapping VPN Rules: Some VPN clients, especially those with split-DNS configurations, might push their own Domains=~. entries, potentially overriding or conflicting with the Consul stub. Ensure the Consul resolver configuration takes precedence.

Verified Configuration Snippets

These are common, tested configurations for various systems.

Linux (`systemd-resolved`)

Create /etc/systemd/resolved.conf.d/10-consul.conf:

[Resolve]
DNS=127.0.0.1:8600
Domains=~consul
DNSSEC=allow-downgrade

Then, restart the service and verify:

sudo systemctl restart systemd-resolved
sudo resolvectl domain # Confirm Global: ~consul

Running Consul on Port 53 (Alternative)

You have two main options if you want clients to query on standard DNS port 53:

Option A – Run Consul on Privileged Port 53: Configure your Consul agent to bind directly to port 53. This often requires running Consul with elevated privileges or using setcap to grant the necessary capabilities.
Option B – Keep Consul on 8600 and Redirect (with iptables): Redirect port 53 traffic to 8600 using iptables rules. This is common for older systemd versions or simpler setups.
```
sudo iptables -t nat -A OUTPUT -d 127.0.0.1 -p udp --dport 53 -j REDIRECT --to-ports 8600
sudo iptables -t nat -A OUTPUT -d 127.0.0.1 -p tcp --dport 53 -j REDIRECT --to-ports 8600
```
Remember to make these iptables rules persistent across reboots.

Command Reference

Tool	Purpose
`dig -p 8600 @127.0.0.1 consul.service.consul`	Directly query the Consul agent's DNS listener to verify its responsiveness.
`resolvectl statistics`	On Linux with `systemd-resolved`, shows cache hits/misses and other resolver stats.
`sudo perf trace -e sendto,recvfrom -p $(pidof consul)`	On Linux, provides detailed syscall tracing to observe packet flow to/from the Consul process.

References