Consul on Kubernetes | Datadog DogstatsD Configuration

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

For customers running Consul versions less than 1.16.x, Datadog DogstatsD integration requires manual configuration for collecting Consul Server-specific runtime metrics. This article outlines the manual steps required to configure a consul-k8s environment for metrics collection via Datadog DogstatsD.

• For users running Consul versions 1.16.x or higher, the Consul helm chart provides datadog-specific overrides, making automating this process easier based on runtime Consul deployment configurations. This method of implementation is recommended for these users in this case.
  • See the Configure Datadog metrics collection for Consul on Kubernetes: DogstatsD documentation for more information regarding this feature.

DogstatsD

This method of integration with Datadog leverages the hashicorp/go-metrics DogstatsD client library to manage metrics collection. Metrics are aggregated and sent via UDP or UDS transports to a Datadog Agent that runs either on the same Kubernetes Cluster as the Consul servers or to an externally accessible agent outside of the Kubernetes Cluster.

Enabling this method of metrics collection allows Consul to control the delivery of metrics traffic directly to a Datadog agent rather than a Datadog agent attempting to reach Consul and scrape the /v1/agent/metrics API endpoint.

This is accomplished by updating each server agent's configuration telemetry stanza and providing the applicable Kubernetes Statefulset labels and annotations for the consul-server Statefulset.

Note on DogstatsD using Unix Domain Sockets:

Leveraging DogstatsD collection using Unix Domain Sockets requires mounting a host directory using Kubernetes volumeMounts in order to share a common socket file for metrics transmission. Use of DogstatsD using Unix Domain Sockets is unsupported for versions of Consul < 1.16.x as available Consul overrides for server.extraVolumes only allows for ConfigMap or secret type volumes. 

This article focuses on UDP Protocol based metrics transmission to the Datadog agent for collection.

Consul Helm Override Configuration

The following Helm value overrides demonstrate how to configure consul-k8s consul-server pod metrics emission to a Datadog Agent leveraging DogstatsD.

After applying the following additional override settings, the users are expected to apply the updates using either the helm upgrade or consul-k8s upgrade commands to update their environments.

Datadog Agent Running in Kube Cluster (namespace: datadog)

Note: Update the address to the accessible Datadog Agent service for your environment. Reference DNS for Services and Pods to determine the accurate agent service name accessible from the consul-server:consul containers.

# File: values.yaml (helm overrides)
server:
  enabled: true
  replicas: 3
  extraLabels:
    'tags.datadoghq.com/version': '1.6.2-ent'
    'tags.datadoghq.com/env': 'vault'
    'tags.datadoghq.com/service': 'consul-server'
  annotations:
    'ad.datadoghq.com/consul.logs': '[{"source": "consul","consul_service": "consul-server"}]'
    'ad.datadoghq.com/consul.metrics_exclude': true
  extraConfig: |
    {
      "telemetry": {
        "prometheus_retention_time": "10s",
        "disable_hostname": true,
        "prefix_filter": ["+consul.rpc.server.call"],
        "enable_host_metrics": true,
        "dogstatsd_tags": ["source:consul","service:consul-server"],
        "dogstatsd_addr": "datadog-agent.datadog.svc.cluster.local"
      }
    }

Datadog Agent Running in Outside Kube Cluster

Note: IP/Hostname is assumed to be routable from Consul servers residing within the Kubernetes cluster.

# File: values.yaml (helm overrides)
# Alternative example - DNS Name: "datadog-agent.my-domain.com:8125"
server:
  enabled: true
  replicas: 3
  extraLabels:
    'tags.datadoghq.com/version': '1.6.2-ent'
    'tags.datadoghq.com/env': 'vault'
    'tags.datadoghq.com/service': 'consul-server'
  annotations:
    'ad.datadoghq.com/consul.logs': '[{"source": "consul","consul_service": "consul-server"}]'
    'ad.datadoghq.com/consul.metrics_exclude': true
  extraConfig: |
    {
      "telemetry": {
        "prometheus_retention_time": "10s",
        "disable_hostname": true,
        "prefix_filter": ["+consul.rpc.server.call"],
        "enable_host_metrics": true,
        "dogstatsd_tags": ["source:consul","service:consul-server"],
        "dogstatsd_addr": "172.20.180.10:8125"
      }
    }

Consul Server Helm Override Amplifying Information

The discussion points below describe the basis for which the override value update is necessary or preferred when configuring DogstatsD metrics collection.

• server.extraLabels: Configures Datadog Unified Service Tagging
  • These tags enable users to:
    • ID deployment impact with trace and container metrics filtered by version.
    • Navigate seamlessly across traces, metrics, and logs with consistent tags.
    • View service data based on environment or version in a unified fashion.
• server.annotations: 
  • 'ad.datadoghq.consul.logs': Configures tags that enable Kubernetes log collection for Consul Servers.
  • 'ad.datadoghq.com/consul.metrics_exclude': When true, excludes metrics collection from the entire consul-server pod.
    • Required to ensure Datadog doesn't leverage Prometheus/Openmetrics style scrape of Consul that could result in configuration conflict when using DogstatsD.
    • DogstatsD, Consul Integration, and Openmetrics Prometheus Metrics by design share the same metric name syntax for collection, and would therefore cause a conflict. The consul.py integration source code prohibits the enablement of more than one integration at a time.
• server.extraConfig (Telemetry):
  • dogstatsd_tags: This provides a list of global tags that will be added to all telemetry packets sent to DogstatsD.
  • dogstatsd_addr: This provides the address of a DogstatsD instance in the format host:port to enable Consul to send various telemetry information to that instance for aggregation.
  • disable_hostname: When true, stops hostname prepending for Datadog gauge-type metrics (simplifies troubleshooting).
  • enable_host_metrics: When true, enables reporting of host metrics about system resources (helpful in troubleshooting)
  • prefix_filter: consul.rpc.server.call filter admittance enables emitting Consul Server Workload metrics (v1.12.x and above only).

References:

1. Consul Agent Telemetry Documentation
2. Datadog DogstatsD
3. Datadog Consul Integration
4. Datadog Kubernetes Prometheus and OpenMetrics metrics collection
5. Datadog integrations-core Source Code
6. HashiCorp go-metrics Source Code