The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

When facing connectivity challenges within your Consul service mesh, the consul-k8s CLI offers powerful troubleshooting tools. This guide consolidates best practices for diagnosing issues across key components, including Dataplane sidecar proxies, the Consul API, and Terminating, Mesh, and Ingress gateways.

Prerequisites

Installing consul-k8s CLI

To utilize the consul-k8s CLI, install it on your Kubernetes/OpenShift management machine, such as your laptop or a jumpbox.

Follow the Install the CLI instructions for your particular management platform distribution.

Kubernetes Contextual Flags

When troubleshooting Consul dataplane and gateway proxies it's important to understand your Kuberenetes/OpenShift environment namespace and context to get the most out of using the consul-k8s CLI tool's proxy subcommand. Ensure you understand for multi-cluster and namespace isolated environments you'll need to be explicit in using the following flags:

Flag	Description
-context	Type: Context String value that sets the Kubernetes context to use for Consul K8s CLI operations.
-kubeconfig	Type: String String value that specifies the path to the kubeconfig file containing applicable cluster contexts/credentials.
-namespace	Type: String Specific namespace pertaining to proxies in question.
-A	Type: Boolean Denotes to parse all namespaces for proxies.

For each of the scenarios described below it's assumed you understand know when/how to apply which of the above context flags t7o troubleshoot the correct pod for a given Kubernetes or OpenShift cluster.

Table of Contents

Troubleshooting pod-to-pod service-mesh connectivity
Troubleshooting pod-to-external service connectivity
Troubleshooting cross-datacenter pod service connectivity
Troubleshooting ingress service-mesh connectivity

Use Cases & Procedures

Troubleshooting pod-to-pod service-mesh connectivity

When you suspect a connectivity issue between two pods both running Consul service mesh sidecars, you can use a combination of the consul-k8s CLI commands to pinpoint and diagnose problems:

Verify that the proxy exists on each pod
Use the consul-k8s proxy list command. For instance, if your pod names include “frontend” and “backend,” ensure that sidecars appear in the output:
```
$ consul-k8s proxy list
Namespace: default

Name                        Type
backend-658b679b45-d5xlb    Sidecar
client-767ccfc8f9-6f6gx     Sidecar
frontend-676564547c-v2mfq   Sidecar
...
```
- If a sidecar proxy is missing, check that Consul’s automatic sidecar injection is enabled for that namespace or that your pod is correctly annotated for sidecar injection.
Inspect the pod’s Envoy configuration
Use consul-k8s proxy read <PODNAME> to confirm that the sidecar is configured with:
- Expected clusters (i.e., the services this sidecar is supposed to communicate with).
- Properly configured listeners and routes.
For instance:
```
$ consul-k8s proxy read backend-658b679b45-d5xlb
...
==> Clusters (5)
Name                    FQDN                                              Endpoints          ...
client                  client.default.dc1.internal.<...>.consul         192.168.18.110:20000 ...
...
```
- If your upstream services (for example, “frontend” or “backend”) are missing from the Clusters section, the Consul configuration may be incomplete or the service registration is incorrect.
- Similarly, check the Listeners or Routes sections to verify the proxy is configured to accept or route traffic as expected.
Check Envoy logs
If configuration looks correct but traffic still fails, you may need to inspect (and possibly adjust) Envoy’s log levels:
```
# Show current log levels
$ consul-k8s proxy log backend-658b679b45-d5xlb

# Increase log level for detailed troubleshooting
$ consul-k8s proxy log backend-658b679b45-d5xlb -update-level debug
```
After adjusting log levels, re-check your application or environment logs to see if any relevant error messages appear, such as Envoy rejecting requests or failing to resolve DNS.
Inspect Envoy cluster stats
Use consul-k8s proxy stats <PODNAME> to view Envoy cluster statistics for that proxy. Look for:
- Circuit breaker state
- Connection failures
- Health check statuses
For example:
```
$ consul-k8s proxy stats backend-658b679b45-d5xlb
cluster.<service>.assignment_stale: 0
cluster.<service>.bind_errors: 0
...
```
If you see non-zero values for errors or circuit breakers, investigate further (e.g., check resource usage, config for limitations, network connectivity, or intentions/ACLs).
Use the troubleshoot subcommands
- consul-k8s troubleshoot upstreams -pod <pod>: Lists all upstreams (both explicit and transparent) that Envoy is configured to connect to from the specified pod.
  If a service you expect to see is missing, you may have missing or incorrect intentions or service definitions.
- consul-k8s troubleshoot proxy -pod <pod> -upstream-ip <ip>: Checks the connectivity from one pod’s Envoy proxy to an upstream’s IP address. The CLI will tell you whether the configuration is valid, the certificates are correct, listeners are present, or if healthy endpoints exist.
Example:
```
$ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-ip 10.4.6.160
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
...
! no healthy endpoints for cluster "backend2.default.dc1.internal..." found
```
This output suggests a missing or unhealthy “backend2” upstream.

Troubleshooting pod-to-external service connectivity

Pods that communicate with external services (for example, external APIs, databases, or SaaS endpoints) rely on Terminating Gateways or direct egress rules in Consul’s configuration:

Check gateway proxies
If using a Terminating Gateway, ensure it appears as a proxy when you run:
```
$ consul-k8s proxy list -A
```
You should see entries of type Terminating Gateway. If missing, the external connectivity may fail due to the gateway not existing or being in a misconfigured state.
Verify gateway config
Use consul-k8s proxy read <pod-running-terminating-gateway> to confirm it has clusters and routes pointing to the external service.
Review logs and stats
- consul-k8s proxy log to adjust or check log levels.
- consul-k8s proxy stats to confirm successful connections or check for circuit breaker openings.
Use troubleshoot commands
If your external service is configured as an upstream in your pods, consul-k8s troubleshoot upstreams -pod <pod> can show if the external IP or domain is recognized. If it’s not recognized, confirm your gateway or external service definition is set up properly in Consul.

Troubleshooting cross-datacenter pod service connectivity

For multi-datacenter or multi-cluster setups, Mesh Gateways are used to route traffic between different Consul clusters. Common steps include:

Check mesh gateway proxies
Run:
```
$ consul-k8s proxy list -A
```
Look for entries of type Mesh Gateway. If these are missing in the relevant namespaces, cross-datacenter connectivity will fail.
Inspect the configuration
Use:
```
$ consul-k8s proxy read <mesh-gateway-pod-name>
```
Confirm it has the clusters or endpoints referencing the remote datacenter’s services.
Check intentions/ACLs
Ensure that the service identity used across datacenters has appropriate ACL tokens/permissions. If there are failing connections or “permission denied” errors in logs, you may have an ACL or intention blocking cross-datacenter traffic.
Troubleshoot from each datacenter
Log in to each cluster’s environment:
- Use consul-k8s troubleshoot proxy or consul-k8s proxy read from both sides to confirm that each gateway knows about the other’s services.

Troubleshooting ingress service-mesh connectivity

Ingress Gateways allow traffic from outside the cluster to enter the Consul service mesh:

Verify ingress gateway existence
Make sure you see an Ingress Gateway in the output of:
```
$ consul-k8s proxy list -A
```
If you do not see your ingress gateway, confirm your Helm values or Kubernetes manifests to ensure it is deployed.
Inspect ingress gateway config
```
$ consul-k8s proxy read <ingress-gateway-pod>
```
Ensure that inbound listeners and routes match the intended external ports and that the correct cluster(s) (destination services) are present in its configuration.
Check logs and stats
As with other proxies:
- Use consul-k8s proxy log <ingress-gateway-pod> to adjust log levels.
- Use consul-k8s proxy stats <ingress-gateway-pod> to view cluster stats, focusing on upstream errors, circuit breaker issues, or any health check failures.
Intentions and ACL checks
Ingress traffic can also fail if intentions are not set to allow traffic from the gateway to the inside service. Verify that you have correct intentions for the ingress gateway’s service identity.

Additional Information

Commands for General Visibility

consul-k8s status
Provides a status summary of the Consul on Kubernetes installation, showing the deployed chart versions, replicas, and health status of servers and clients. Useful for a high-level health check before diving deeper into proxy-level issues.
consul-k8s troubleshoot upstreams and consul-k8s troubleshoot proxy
Both subcommands under the troubleshoot umbrella are particularly helpful for pinpointing misconfigurations or network issues from within a pod’s Envoy sidecar.

Conclusion

Using the consul-k8s command line tool consolidates multiple troubleshooting steps into a single interface. By examining proxy configurations, logs, and network stats, you can methodically isolate whether an issue stems from misconfigurations, intentions and ACLs, cluster connectivity, or gateway health. Combine these CLI steps with standard Kubernetes diagnostics (e.g., checking Pod and Service states, verifying your kubeconfig context/namespace) for a comprehensive troubleshooting workflow.

If you continue to experience issues, confirm you have followed Consul’s official installation guidelines for your environment, and that your pods are annotated/enabled for sidecar injection as required.

Consul-k8s CLI: Service Mesh Troubleshooting