The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

Dynatrace offers extensions for metrics observability for Prometheus endpoints. This article covers what Dynatrace-specific annotations and Consul metrics configurations are required to provide insight to the health and status of Consul Dataplane components.

Consul Prometheus Endpoints for Sidecars

Mesh Metrics Merging with Consul Dataplane

Envoy sidecar metrics and Connect service metrics can both be scraped from a single endpoint known as the Merged Metrics Port. This merged metrics functionality is enabled via either:

• Helm Override Global Setting:connectInject.defaultMergedMetricsPort
• Pod Annotation: consul.hashicorp.com/merged-metrics-port

When metrics and metrics merging are both enabled, metrics are combined from the Connect service (if your service exposes its own Prometheus metrics endpoint) and the Envoy sidecar proxy.

• This endpoint is called the merged metrics endpoint and defaults to 127.0.0.1:20100/stats/prometheus.
• The 127.0.0.1:20100/stats/prometheus endpoint is exposed to the local pod only via 127.0.0.1 and is not routable from Dynatrace ActiveGate.
• The reachable merged metrics listener defaults to the value of 0.0.0.0:20200 where the merged metrics will be available for ActiveGate scraping.

Observing 127.0.0.1:20100/stats/prometheus and 0.0.0.0:20200 endpoint metrics data

You can see that both endpoints for the merged metrics server and exposed listener contain the same metric data by hitting the admin API on the service pod directly (assuming curl is available):

$ curl -s 127.0.0.1:20100/stats/prometheus | head -n5
# HELP consul_dataplane_connect_duration This will be a sample of the time it takes to get connected to a server. This duration will cover everything from making the server features request all the way through to opening an xDS session with a server
# TYPE consul_dataplane_connect_duration summary
consul_dataplane_connect_duration{quantile="0.5"} NaN
consul_dataplane_connect_duration{quantile="0.9"} NaN
consul_dataplane_connect_duration{quantile="0.99"} NaN

$ curl -s 0:20200/metrics | head -n5
# HELP consul_dataplane_connect_duration This will be a sample of the time it takes to get connected to a server. This duration will cover everything from making the server features request all the way through to opening an xDS session with a server
# TYPE consul_dataplane_connect_duration summary
consul_dataplane_connect_duration{quantile="0.5"} NaN
consul_dataplane_connect_duration{quantile="0.9"} NaN
consul_dataplane_connect_duration{quantile="0.99"} NaN

Connect Service Metrics

The Consul Dataplane metrics merging server with reach out to the service's configured Prometheus endpoint set with the pod level annotations:

'consul.hashicorp.com/service-metrics-port': '<your_svc_app_port>'
'consul.hashicorp.com/service-metrics-path': '/metrics'

Whatever data is returned from this endpoint is then aggregated into the overall metric return provided at the expose 0.0.0.0:20200 listener. Your application should be exposing Prometheus styled metrics from this endpoint.

Connect-inject Default Prometheus Annotations

The connect-injector will automatically add prometheus annotations to connect-injected pods. While these are useful for observability providers like Datadog, Dynatrace requires an alternate set of annotations to notify ActiveGate to scrape.

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "20200"

It will also add a listener on the Envoy sidecar to expose metrics (i.e., 0.0.0.0:20200).

• The 0.0.0.0:20200 listener targets the merged metrics endpoint 127.0.0.1:20100/stats/prometheus.
• The exposed listener (0.0.0.0:20200) port may be altered with the above-mentioned Helm value (or overridden on a per-Pod basis with consul.hashicorp.com/merged-metrics-port).

Consul Service Mesh Sidecars and Dynatrace

At a minimum, we need to be able to identify and point Dynatrace to the correct listener. 

ConnectInject Metrics Configuration (Helm Overrides)

The below Helm Chart override values will globally enable a default listener to be exposed on all Dataplane Sidecar Proxies.

connectInject:
  metrics:
    defaultEnabled: true
    defaultEnableMerging: true
    defaultMergedMetricsPort: '20100'
    defaultPrometheusScrapePort: '20200'
    defaultPrometheusScrapePath: '/metrics'

Pod Level Service Mesh Configuration (Annotations)

The sidecar metrics can also be configured on a per-pod basis. The below demonstrates the equivalent settings that would be set from the previously demonstrated Helm Chart overrides.

annotations:
  'consul.hashicorp.com/enable-metrics': 'true'
  'consul.hashicorp.com/enable-metrics-merging': 'true'
  'consul.hashicorp.com/prometheus-scrape-port': '20200'
  'consul.hashicorp.com/prometheus-scrape-path': '/metrics'
  'consul.hashicorp.com/merged-metrics-port': '20100'
  'consul.hashicorp.com/merged-metrics-path': '/stats/prometheus'
  'consul.hashicorp.com/service-metrics-port': '<my_app_port>'
  'consul.hashicorp.com/service-metrics-path': '/metrics'

• Once you decide on how to configure your Consul Dataplane, either globally or pod-specific, you may proceed to determine where to point Dynatrace to scrape and retrieve metrics.

Determining Dynatrace Annotation Port and Path Values

Observing the deployed listeners of a Dataplane Sidecar with the above sidecar metrics configurations we can see:

# Consul Dataplane Sidecar Listeners

$ curl -s 0:19000/listeners
  envoy_prometheus_metrics_listener::0.0.0.0:20200
  envoy_metrics_listener::127.0.0.1:21200
  public_listener:10.129.2.35:20000::10.129.2.35:20000
  outbound_listener:127.0.0.1:15001::127.0.0.1:15001

Focusing on the Envoy metrics listeners specifically, we note:

• envoy_prometheus_metrics_listener: 0.0.0.0:20200
  • This is listener exposed for collection of merged metrics as retrieved from the Dataplane merged metrics endpoint. This essentially reflects the 127.0.0.1:20100/stats/prometheusendpoint separately to not allow exposing the metrics server directly.
  • This is the port of interest to Dynatrace.
• envoy_metrics_listener: 127.0.0.1:21200
  • Listener created for exposing Envoy /statsendpoint for Envoy specific metrics.
  • This endpoint is inaccessible from any external entities of the pod, so this isn't of interest here.

So what do we use?

• Dynatrace Annotation Port Value: 20200
• Dynatrace Annotation Path Value: /metrics

Configure/Update Pods for Dynatrace Prometheus Scrape

Following the Dynatrace documentation for Monitoring Prometheus on Kubernetes, the service mesh enabled pod annotations are as follows (add these in addition to the HashiCorp specific ones if using per-pod annotations for enabling Dataplane sidecar metrics):

annotations:
  'metrics.dynatrace.com/scrape': 'true'
  'metrics.dynatrace.com/port': '20200'
  'metrics.dynatrace.com/path': '/metrics'
  'metrics.dynatrace.com/secure': 'false'

Enable Dynatrace Prometheus Pod Annotation Monitoring

We'll also need to ensure the Dynatrace ActiveGate is configured to monitor Prometheus annotated pods. From your Dynatrace environment web portal ensure the Settings for Cloud and Virtualization > Kubernetes have the Monitor annotated Prometheus exporters set to enabled.

• From here metrics should be available from your Dynatrace environment portal.

Verify Envoy and Service Metrics Collection

You should now be able to observe Envoy and whatever application specific metrics being collected from the exposed Prometheus endpoint from your Dynatrace Metrics explorer. Search for envoyto verify.