Dynatrace: Consul K8s Prometheus Configuration

Avatar
Nathan Mollica
  • Updated

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

This guide outlines the steps required to configure Dynatrace for monitoring Consul service mesh deployments on Kubernetes. By leveraging Prometheus annotations and enabling specific settings within Dynatrace, you can gain comprehensive observability into the performance and health of both your Envoy sidecar proxies and application services. This integration allows you to collect and visualize key metrics, enabling proactive monitoring, efficient troubleshooting, and optimized performance management for your Consul-based microservices.

Expected Outcome

The expected outcome of configuring Dynatrace Annotations for Prometheus monitoring is to enable Dynatrace to collect and visualize metrics from your Consul service mesh, specifically from Envoy sidecars and application services.

Prerequisites 

  • Consul 1.10 or later: Ensure you have a compatible version of Consul installed, as this integration leverages features available in Consul 1.10 and later.
  • Consul Connect Injector: The Connect Injector is responsible for injecting the Envoy sidecar and configuring the necessary annotations for Dynatrace. Verify that it's properly deployed and configured in your Kubernetes cluster.
  • Consul-K8s: This is the official HashiCorp project for running Consul on Kubernetes. You'll need it for managing and integrating Consul with your Kubernetes environment.
  • Kubernetes Environment:
    • Kubernetes Cluster: A running Kubernetes cluster is essential for deploying your applications and the Consul service mesh.
    • kubectl: The Kubernetes command-line tool, kubectl, is necessary for interacting with your cluster and managing resources.
    • Helm (Optional): If you're using Helm to manage Consul deployments, ensure you have Helm installed and configured.
  • Dynatrace Setup:
    • Dynatrace Account: An active Dynatrace account with appropriate permissions to configure Kubernetes monitoring and access metrics.
    • Dynatrace OneAgent: The Dynatrace OneAgent needs to be installed in your Kubernetes cluster to collect metrics and other observability data.
    • Kubernetes Settings: Ensure that Dynatrace is configured to monitor your Kubernetes cluster. This typically involves setting up a connection between Dynatrace and your cluster.

Use Case

This integration between Dynatrace and Consul is particularly valuable in scenarios where you need deep insights into your Consul service mesh running on Kubernetes.

 

Table of Contents

 

Procedure

Configuring Dynatrace Annotations for Prometheus Monitoring

Dynatrace requires specific annotations to enable Prometheus metric scraping. These annotations should be added to the relevant Kubernetes resources (such as Consul proxy sidecars) alongside any HashiCorp-specific configurations.

Required Annotations:

annotations:
  metrics.dynatrace.com/scrape: "true"
  metrics.dynatrace.com/port: "20200"
  metrics.dynatrace.com/path: "/metrics"
  metrics.dynatrace.com/secure: "false"

These annotations signal Dynatrace to scrape metrics from the specified port (20200) and endpoint (/metrics).

 

Enabling Dynatrace to Monitor Annotated Prometheus Exporters

To ensure that Dynatrace collects Prometheus metrics from annotated Kubernetes pods, follow these steps:

  1. Access Dynatrace Portal: Log into your Dynatrace web portal.
  2. Navigate to Kubernetes Settings:
    • Go to Settings > Cloud and Virtualization > Kubernetes.
  3. Enable Prometheus Exporter Monitoring:
    • Locate the option Monitor annotated Prometheus exporters and enable it.

This setting ensures that Dynatrace will automatically discover and scrape metrics from annotated pods.

 

Verifying Metrics Collection

After configuring the annotations and enabling Prometheus monitoring in Dynatrace, verify the integration by checking the collected metrics.

Step 1: Use Dynatrace Metrics Explorer

  1. Navigate to the Metrics Explorer in the Dynatrace portal.
  2. Search for envoy to view collected Envoy proxy and application-specific metrics.

Step 2: Validate Envoy Listeners for Prometheus Metrics

Use the following command to check the active listeners on the Dataplane sidecar:

curl -s 0:19000/listeners

Key listeners to verify:

  • Merged Metrics Listener:

    envoy_prometheus_metrics_listener::0.0.0.0:20200

    This is the primary endpoint used by Dynatrace for Prometheus metric scraping.

  • Envoy Stats Listener:

    envoy_metrics_listener::127.0.0.1:21200

    This listener provides additional Envoy-related metrics.

If the 20200 port is active, Dynatrace should be able to collect Prometheus metrics successfully.

 

Configuring Consul-K8s for Prometheus Observability in Helm

To enable Prometheus metric scraping for Consul-K8s in Helm, update the Consul Helm chart values with the following configurations:

Step 1: Enable Prometheus Metrics in Consul Helm Values

Modify your Helm values.yaml file to include:

connectInject:
  metrics:
    defaultEnabled: true
    defaultPrometheusScrapePort: "20200"
    defaultPrometheusScrapePath: "/metrics"

This ensures that all injected Consul sidecars expose Prometheus metrics for Dynatrace.

Step 2: Apply the Updated Helm Values

Run the following command to apply the updated configuration:

helm upgrade --install consul hashicorp/consul -f values.yaml

This will redeploy Consul with the necessary annotations for Prometheus metric collection by Dynatrace.

 

Conclusion

By following this guide, you can successfully enable Dynatrace to monitor Consul-K8s metrics via Prometheus scraping. Ensure that the correct annotations are applied, ActiveGate settings are enabled, and Envoy listeners are verified. With these configurations in place, Dynatrace will provide comprehensive insights into Consul Dataplane health and performance.

For further troubleshooting, refer to Dynatrace’s Prometheus integration documentation or HashiCorp’s Consul monitoring guidelines.

 

Additional Information

Articles in this section

Related articles