How to Enable Metrics from a Gateway Sidecar to Datadog on VMs

Avatar
Bruce Lok
  • Updated

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

 

Introduction

This article discusses how to enable metrics from Envoy sidecars to Datadog originating from gateways such as the mesh gateway and ingress gateway. Envoy sidecars are a common way to provide a number of features to services, such as load balancing, traffic routing, and security. Datadog is a cloud-based monitoring and analytics platform that can be used to collect and visualize metrics from a variety of sources.

 

Expected Outcome

Once you have completed these steps, you will see the metrics are being sent to Datadog by using the Datadog UI. The metrics from Envoy sidecars appear under the Infrastructure tab.

Prerequisites

To enable metrics from Envoy sidecars to Datadog, you will need to:

  • Install a Datadog agent on your VMs. The Datadog agent is a software component that collects metrics from your VMs and sends them to Datadog.
  • Configure the Datadog agent to listen for UDP port 8125. This is the port that Envoy sidecars will use to send metrics to Datadog.

 

Use Case

This is an article about transmitting Consul and Envoy metrics to Datadog. It discusses how to enable metrics and push them to Datadog. It also provides instructions on how to configure proxy-default. Some of the important points from this article are that the configuration will only take effect for newly started Envoy sidecars.

Procedure

Push Consul metrics to the Datadogd

To enable Consul to push metrics directly to the Datadog Agent as a telemetry sink, you need to provide the address of this service in the Consul configuration.

  1. Add the following configuration block to each Consul agent.
    "telemetry": {
          "dogstatsd_addr": "127.0.0.1:8125"
         },
  2. Restart the agent for the changes to take effect.
    • After the restart, in addition to the default in-memory sink and any other metrics that are enabled, the Consul agent will begin sending its generated metrics to DogStatsD.

 

Enable exposing metrics from the Envoy sidecar.

The previous step is an initial step to expose metrics from the Consul agent, but it does not include metrics from the Envoy sidecar within the Consul agent. In order to direct Envoy on where to push the metrics, there are several methods available.

 

Configure proxy-default

In order to instruct Envoy on where to push the metrics, we can add a configuration block to the proxy-default that includes the envoy_dogstatsd_url.

  1. Add the below to the proxy-default configuration.
    • This is a global configuration so it will apply to all new Envoy sidecars when they bootstrap.
      # proxy-default.hcl
      Kind = "proxy-defaults"
      Name = "global"
      Config {
        envoy_dogstatsd_url = "udp://127.0.0.1:8125"
      }
  2. Apply the configuration
    $ consul config write proxy-default.hcl
  3. Start an Envoy sidecar using consul connect envoy
    $ consul connect envoy -sidecar-for dashboard -bootstrap | grep -i dogstatsd -B4 -A10
      "stats_sinks": [
        {
          "name": "envoy.stat_sinks.dog_statsd",
          "typedConfig": {
            "@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
            "address": {
              "socket_address": {
                "address": "127.0.0.1",
                "port_value": 8125
              }
            }
          }
        }
      ],

    • This configuration will be parsed to Envoy's bootstrap configuration, meaning that it will only take effect for newly started Envoy sidecars.
    • Verify this by using the -bootstrap option to generate the Envoy bootstrap configuration. In the above example, the Envoy sidecar was started for a service named Dashboard.
      • We see that the Datadog URL was successfully injected into Envoy's bootstrap configuration.

 

Ingress Gateway

The proxy-defaults option envoy_dogstatsd_url applies to the ingress-gateway as well, because it is a type of proxy that is registered as a service in Consul.

  • Once defined, start the ingress-gateway using the following command, which also registers it as a service. 
     
    $ consul connect envoy -gateway=ingress -register \
          -service ingress-service \
          -address '{{ GetInterfaceIP "eth0" }}:8888'

 

Mesh Gateway

Apart from configuring the proxy-default config entries, It is possible to enable mesh-gateway to push metrics from Envoy to StatsD, it should begin by service registration of mesh-gateway.  The config envoy_dogstatsd_url can be parsed through service registration for mesh-gateway, enabling the metrics to be pushed to StatsD.

Below is an example of how this can be achieved.

  1. Register the service mesh-gateway with the below context in the VM host.
     
    # mesh-gw.hcl
    service {
      name = "mesh-gw"
      kind = "mesh-gateway"
      proxy {
        config = {
          envoy_dogstatsd_url = "udp://127.0.0.1:8125"
        }
      }
      port = 8443
    }
  2. Verify that the envoy_dogstatsd_url parameter was populated within the service definition by making a request to the API endpoint.
     
    $ curl -s http://127.0.0.1:8500/v1/catalog/service/mesh-gw?pretty
    ...
       "ServiceProxy": {
          "Mode": "",
          "Config": {
            "envoy_dogstatsd_url": "udp://127.0.0.1:8125"
           },
           "MeshGateway": {
           "Mode": "local"
           },
           "Expose": {}
       },
    ...
  3. Do a dry run to validate that the configurations will parse to Envoy's bootstrap configuration by adding -bootstrap. 
    • In this example, we can see the entry envoy.stat_sinks.dog_statsd with an address of 127.0.0.1 and port 8125 are appended in the bootstrap configuration. 
       
      $ consul connect envoy -gateway=mesh -service=mesh-gw -bootstrap | grep -i dogstatsd -B4 -A10
      "stats_sinks": [
        {
          "name": "envoy.stat_sinks.dog_statsd",
          "typedConfig": {
            "@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
            "address": {
              "socket_address": {
                "address": "127.0.0.1",
                "port_value": 8125
              }
            }
          }
        }
      ],
      "stats_config": {
  4. Start Envoy alongside the mesh gateway and verify that the metrics appear in Datadog.
     
    $ consul connect envoy -gateway=mesh -service=mesh-gw

Reference:

Articles in this section

See more

Related articles