Enable Metrics from a Gateway Sidecar to Datadog on VMs

Avatar
Bruce Lok
  • Updated

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

 

Introduction

This page provides insights on effectively transmitting Consul and Envoy metrics to Datadog, with a specific focus on Envoy sidecar metrics originating from gateways such as the mesh gateway and ingress gateway.

 

Prerequisites

 

Instructions

Push Consul metrics to the Datadogd

To enable Consul to push metrics directly to the Datadog Agent as a telemetry sink, you need to provide the address of this service in the Consul configuration.

  1. Adding the following configuration block to each Consul agent.
    "telemetry": {
          "dogstatsd_addr": "127.0.0.1:8125"
         },
  2. Restart the agent for the changes to take effect.
    • After the restart, in addition to the default in-memory sink and any other metrics that are enabled, the Consul agent will begin sending its generated metrics to DogStatsD.

 

Enable exposing metrics from the Envoy sidecar.

The previous step is an initial step to expose metrics from the Consul agent, but it does not include metrics from the Envoy sidecar within the Consul agent. In order to direct Envoy on where to push the metrics, there are several methods available.

 

Configure proxy-default

In order to instruct Envoy on where to push the metrics, we can add a configuration block to the proxy-default that includes the envoy_dogstatsd_url.

  1. Add the below to the proxy-default configuration.
    • This is a global configuration so it will apply to all new Envoy sidecars when they bootstrap.
      # proxy-default.hcl
      Kind = "proxy-defaults"
      Name = "global"
      Config {
        envoy_dogstatsd_url = "udp://127.0.0.1:8125"
      }
  2. Apply the configuration
    $ consul config write proxy-default.hcl
  3. Start an Envoy sidecar using consul connect envoy
    $ consul connect envoy -sidecar-for dashboard -bootstrap | grep -i dogstatsd -B4 -A10
      "stats_sinks": [
        {
          "name": "envoy.stat_sinks.dog_statsd",
          "typedConfig": {
            "@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
            "address": {
              "socket_address": {
                "address": "127.0.0.1",
                "port_value": 8125
              }
            }
          }
        }
      ],

    • This configuration will be parsed to Envoy's bootstrap configuration, meaning that it will only take effect for newly started Envoy sidecars.
    • Verify this by using the -bootstrap option to generate the Envoy bootstrap configuration. In the above example, the Envoy sidecar was started for a service named Dashboard.
      • We see that the Datadog URL was successfully injected into Envoy's bootstrap configuration.

 

Ingress-Gateway

The proxy-defaults option envoy_dogstatsd_url applies to the ingress-gateway as well, because it is a type of proxy that is registered as a service in Consul.

  • Once defined, start the ingress-gateway using the following command, which also registers it as a service. 
     
    $ consul connect envoy -gateway=ingress -register \
          -service ingress-service \
          -address '{{ GetInterfaceIP "eth0" }}:8888'

 

 

Mesh-Gateway

Apart from configuring the proxy-default config entries, It is possible to enable mesh-gateway to push metrics from Envoy to StatsD, it should begin by service registration of mesh-gateway.  The config envoy_dogstatsd_url can be parsed through service registration for mesh-gateway, enabling the metrics to be pushed to StatsD.

Below is an example of how this can be achieved.

  1. Register the service mesh-gateway with the below context in the VM host.
     
    # mesh-gw.hcl
    service {
      name = "mesh-gw"
      kind = "mesh-gateway"
      proxy {
        config = {
          envoy_dogstatsd_url = "udp://127.0.0.1:8125"
        }
      }
      port = 8443
    }
  2. Verify that the envoy_dogstatsd_url parameter was populated within the service definition by making a request to the API endpoint.
     
    $ curl -s http://127.0.0.1:8500/v1/catalog/service/mesh-gw?pretty
    ...
       "ServiceProxy": {
          "Mode": "",
          "Config": {
            "envoy_dogstatsd_url": "udp://127.0.0.1:8125"
           },
           "MeshGateway": {
           "Mode": "local"
           },
           "Expose": {}
       },
    ...
  3. Do a dry run to validate that the configurations will parse to Envoy's bootstrap configuration by adding -bootstrap. 
    • In this example, we can see the entry envoy.stat_sinks.dog_statsd with an address of 127.0.0.1 and port 8125 are appended in the bootstrap configuration. 
       
      $ consul connect envoy -gateway=mesh -service=mesh-gw -bootstrap | grep -i dogstatsd -B4 -A10
      "stats_sinks": [
        {
          "name": "envoy.stat_sinks.dog_statsd",
          "typedConfig": {
            "@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
            "address": {
              "socket_address": {
                "address": "127.0.0.1",
                "port_value": 8125
              }
            }
          }
        }
      ],
      "stats_config": {
  4. Start Envoy alongside the mesh-gateway and verify that the metrics appear in Datadog.
     
    $ consul connect envoy -gateway=mesh -service=mesh-gw

Reference:

Articles in this section

See more

Related articles