How-to collect Envoy sidecar debug logs and API endpoint outputs from Nomad workloads – HashiCorp Help Center

Introduction

This is to provide an example on how to collect Envoy debug logs and API endpoint outputs, while Envoy sidecar is running on a Nomad workloads.

Expected Outcome

Being able to collect/execute Envoy API calls as such as:

# enable envoy's debug logging and reset counters
curl -XPOST http://127.0.0.2:19001/reset_counters
curl -XPOST http://127.0.0.2:19001/logging?level=debug

# scrape envoy API endpoints 
curl http://127.0.0.2:19002/stats?pretty
curl http://127.0.0.2:19002/listeners?pretty
curl http://127.0.0.2:19002/clusters?pretty
curl http://127.0.0.2:19002/config_dump?include_eds

Prerequisites

Envoy sidecar task does not have wget and curl utilities installed, hence we need to use debug task in order to access Envoy API.
We need first to find Envoy's listening ip_address:port, you can check this by going to the Nomad's UI -> Job -> <job_name> -> Allocations -> connect-proxy-<job_name> -> View Logs -> stdout/stderr, and look for Envoy message similar to [info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.2:19002, where admin address is the address:port where envoy is listening at and against which you should query the Envoy API endpoints.
Note that the debug sidecar task should be deployed under the targeted task group, so you can have access to the local task group where envoy resides.

Use Case

Procedure

Insert debug task into the task group running the envoy sidecar in question:

    task "debug-api" {
      driver = "docker"
      config {
        image = "nicolaka/netshoot"
        command = "/bin/bash"
        args = ["-c", "while true; do ping localhost; sleep 60;done"]
     }
   }

Exec into debug task (debug-api) and execute curl/wget against envoy's sidecar admin address:

nomad alloc exec -i -t -tls-skip-verify -token <acl_token> -task debug-api <alloc_id> sh
# enable envoy debug logging level
curl --request POST http://127.0.0.2:19002/logging?level=debug
# clear envoy counters
curl --request POST http://127.0.0.2:19002/reset_counters
# collect and re-direct envoy stats API endpoint output
curl http://127.0.0.2:19002/stats > envoy_stats.log

Complete job example can be found below:

job "countdash" {

  group "api" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"

     meta {
       test = "1"
    }
      connect {
        sidecar_service {}
      }
    }

    task "web" {
      driver = "docker"
      config {
        image          = "hashicorpdev/counter-api:v3"
        auth_soft_fail = true
      }
    }

    task "debug-api" {
      driver = "docker"
     config {
        image = "nicolaka/netshoot"
        command = "/bin/bash"
        args = ["-c", "while true; do ping localhost; sleep 60;done"]
     }
   }

  group "dashboard" {
   network {
     mode = "bridge"
     port "http" {
       static = 9002
       to     = 9002
     }
   }

    service {
     name = "count-dashboard"
     port = "9002"
     connect {
       sidecar_service {
         proxy {
           upstreams {
             destination_name = "count-api"
             local_bind_port  = 8080
           }
         }
       }
     }
   }

    task "dashboard" {
     driver = "docker"
     env {
       COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
     }
     config {
       image          = "hashicorpdev/counter-dashboard:v3"
       auth_soft_fail = true
     }
   }
    task "debug-api" {
      driver = "docker"
     config {
        image = "nicolaka/netshoot"
        command = "/bin/bash"
        args = ["-c", "while true; do ping localhost; sleep 60;done"]
    }
  }
 }
}

Additional Information