Expected Outcome
Envoy metrics will be accessible with Consul running on Kubernetes.
Primer
- Envoy: A service proxy for communication between applications within a service mesh.
Use Case
There are times when Consul's service mesh is showing connectivity issues, which can correlate to Envoy and the network. An important part of observability is retrieving all relevant metrics to understand the bigger picture. This article will show operators how to access Envoy metrics for troubleshooting issues.
Procedure
Accessing Envoy metrics from a Kubernetes pod
- kubectl port-forward pod/counting-774d99898d-skkgp 19000
If the pod is in a namespace(like consul)
- kubectl port-forward pod/counting-774d99898d-skkgp -n consul 19000
From the browser, the operator can then go to localhost:19000/stats, or localhost:19000/clusters to see the cluster stats. If for some reason accessing the browser isn't possible, the operator can curl from the terminal:
curl --location --request GET 'http://localhost:19001/stats?filter=^cluster.<service-name>'
For example, querying stats for the counting service(from the dashboard sidecar) would look like the following:
curl --location --request GET 'http://localhost:19001/stats?filter=^cluster.counting'
Truncated Output:
cluster.counting.default.dc1.internal.b07f5740-6c72-36f9-b101-c90a9e878ac3.consul.version_text: ""
cluster.counting.default.dc1.internal.b07f5740-6c72-36f9-b101-c90a9e878ac3.consul.assignment_stale: 0
cluster.counting.default.dc1.internal.b07f5740-6c72-36f9-b101-c90a9e878ac3.consul.assignment_timeout_received: 0
cluster.counting.default.dc1.internal.b07f5740-6c72-36f9-b101-c90a9e878ac3.consul.bind_errors: 0
Now the operator can curl for specific stats with this information:
curl --location --request GET 'http://localhost:19001/stats?filter=^cluster.counting.default.dc1.internal.b07f5740-6c72-36f9-b101-c90a9e878ac3.consul.upstream_rq_total$'
Output:
cluster.counting.default.dc1.internal.b07f5740-6c72-36f9-b101-c90a9e878ac3.consul.upstream_rq_total: 4893
Additional information
This article will assist in accessing Envoy metrics. To get a better understanding on how Envoy metrics relates to Consul's service mesh, refer here.