The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
Introduction
Observing Consul service mesh metrics in Kubernetes is critical for monitoring the health and performance of your services. This guide walks you through configuring Helm overrides and pod annotations required to enable Prometheus metric collection for Consul sidecars and services.
Expected Outcome
Enable Prometheus monitoring for your Consul service mesh. This involves exposing metrics from both Envoy sidecars and your application services, making them accessible for scraping and analysis by Prometheus.
Prerequisites
- Consul 1.10 or later: The merged metrics feature is available from Consul 1.10 onwards. Ensure you have a compatible version installed.
- Consul Connect Injector: The Connect Injector is responsible for injecting the Envoy sidecar and configuring the necessary annotations for Prometheus. Verify that it's properly deployed and configured in your Kubernetes cluster.
- Consul-K8s: This is the official HashiCorp project for running Consul on Kubernetes. You'll need it for managing and integrating Consul with your Kubernetes environment.
-
Kubernetes Environment:
- Kubernetes Cluster: A running Kubernetes cluster is essential for deploying your applications and the Consul service mesh.
-
kubectl: The Kubernetes command-line tool,
kubectl, is necessary for interacting with your cluster and managing resources. - Helm (Optional): If you're using Helm to manage Consul deployments, ensure you have Helm installed and configured.
-
Prometheus Setup:
- Prometheus Server: A running Prometheus server is required to scrape and store the metrics exposed by your Consul service mesh.
-
Prometheus Configuration: Configure Prometheus to scrape the appropriate endpoints (e.g.,
0.0.0.0:20200) and any other relevant targets in your Consul environment.
Use Case
To gain comprehensive visibility into the health and performance of your services and infrastructure.
Table of Contents
- Introduction
- Configuring Connect-Inject Default Prometheus Annotations
- Consul Service Mesh Sidecar Metrics Configuration
- Application Merged Metrics Endpoints for Sidecars
- Verifying Metrics Collection
- Conclusion
- References
Procedure
Configuring Connect-Inject Default Prometheus Annotations
By default, Consul Connect Injector adds Prometheus annotations to injected pods. When the applicable Helm Chart Values or Pod-Specific Annotations are configured (described further in the next section)
Default Prometheus Annotations (Injected via Helm)
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "20200"
These ensure Prometheus scrapers can collect metrics from Consul sidecars.
Consul Service Mesh Sidecar Metrics Configuration
Global Configuration via Helm
To enable global metric merging for all Consul dataplane proxies, modify your Helm chart as follows:
connectInject:
metrics:
defaultEnabled: true
defaultPrometheusScrapePort: "20200"
defaultPrometheusScrapePath: "/metrics"
This configuration ensures that every injected Consul sidecar proxy exposes Prometheus metrics on 0.0.0.0:20200.
Pod-Specific Configuration
For granular control at the pod level, use the following annotations:
annotations:
consul.hashicorp.com/enable-metrics: "true"
consul.hashicorp.com/prometheus-scrape-port: "20200"
consul.hashicorp.com/prometheus-scrape-path: "/metrics"
These annotations ensure that both Envoy stats are accessible via Prometheus scrapers.
Application Merged Metrics Endpoints for Sidecars
Merging Envoy Sidecar and Application Service Metrics
Consul allows you to merge metrics from the Envoy sidecar and the Connect App's service into a single Prometheus endpoint. This can be enabled using:
Helm Global Override:
connectInject:
metrics:
defaultEnabled: true
defaultEnableMerging: true
# The port at which the consul-dataplane will listen on to return
# merged metrics stats.
defaultMergedMetricsPort: 20100Pod Annotation:
The annotations below demonstrate per-pod annotation overrides for the above Helm chart value equivalent settings.
annotations:
consul.hashicorp.com/enable-metrics: true
consul.hashicorp.com/enable-metrics-merging: true
consul.hashicorp.com/merged-metrics-port: "20100"
consul.hashicorp.com/merged-metrics-path: "/stats/prometheus"Application Service Metrics Configuration
To collect Prometheus metrics from your application service, you must define the application's metrics endpoint using pod-level annotations only:
annotations:
consul.hashicorp.com/service-metrics-port: "<your_service_port>"
consul.hashicorp.com/service-metrics-path: "<your_service_metrics_api_endpoint>"
These metrics will be aggregated and exposed at 0.0.0.0:20200, ensuring both Envoy and service metrics are available in a unified view.
Verifying the Merged Metrics Endpoints:
When enabled, the merged metrics are exposed at:
-
Local (within pod):
127.0.0.1:20100/stats/prometheus -
External scraping endpoint:
0.0.0.0:20200
curl -s 127.0.0.1:20100/stats/prometheus | head -n 5
curl -s 0.0.0.0:20200/metrics | head -n 5
Both endpoints should return the same metrics data.
Verifying Metrics Collection
Validate Envoy Sidecar Listeners
curl -s 0.0.0.0:19000/listeners
Key listeners to verify:
-
Merged Metrics Listener:
envoy_prometheus_metrics_listener::0.0.0.0:20200 -
Envoy Stats Listener:
envoy_metrics_listener::127.0.0.1:20100
Conclusion
With these configurations, you can successfully integrate Consul-K8s for Prometheus-based monitoring. These settings ensure comprehensive visibility into both Envoy proxy metrics and application service metrics, facilitating proactive service health monitoring within a Consul service mesh environment.