Accessing and setting Envoy logs for Consul

Avatar
Mark Campbell-Vincent
  • Updated

Expected Outcome

Envoy logs will be accessible and ready for debugging.

Primer

- Envoy: A service proxy for communication between applications within a service mesh.

- upstream: the upstream cluster in Envoy that will be receiving requests. The envoy logs will show any request/connect timeouts errors with upstream in debug mode

- http: any http request made through Envoy will be recorded here, including http requests to sidecar proxies or the Envoy admin UI.

- config: the config component is meant to collect any configuration updates to Envoy clusters(dynamic listeners, routers, clusters, and endpoints). Setting this in debug mode will reflect these changes.

- router: the router takes an incoming HTTP request, matches it to an upstream cluster, acquires a connection pool to a host in the upstream cluster, and forwards the request.

Use Case

There are times when services registered in Consul are not reachable, and observing logs for those services are invaluable in that moment. When connect is enabled, services have a sidecar proxy that leverages the Envoy proxy by default. Since service-to-service communication occurs through Envoy sidecar proxies, Envoy logs can shed light on errors between services. There are a few ways to access Envoy logs and set log levels to start debugging. In this article, we will be going over this with and without the Consul Helm Chart.

Procedure

Setting Envoy logs in the Helm configuration

The Consul helm chart uses envoyExtraArgs: to leverage Envoy command line options. One of the helpful options is --component-log-level. This provides granular control over setting log levels for Envoy components. In the example below, the components upstream, http, router and config are set to the debug log level. These four components are vital when debugging issues with requests between your services(sidecar proxies).

connectInject:
  enabled: true
  envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"
Once added in the values.yaml files, upgrade helm(e.g. helm upgrade consul hashicorp/consul -f val.yaml). After the helm upgrade, it  is necessary to restart the pods(services) to be able to see envoy logs in debug/trace mode. Take note that restarting pods can have an intermittent disruption in reaching services, so planning a maintenance window is recommended. 

If you haven't set envoyExtraArgs: in consul-values.yaml just yet, you can set the log levels on the fly by using the following kubectl command:

kubectl exec pod/<pod-name> -c <container-name> -- curl -X POST http://localhost:19000/logging?config=debug
Example:
kubectl exec pod/static-client-5bf4575d9c-zr2b -c static-client -- curl -X POST  http://localhost:19000/logging?config=debug

You can execute the kubectl command for each component or execute for multiple in one command:

kubectl exec pod/static-client-5bf4575d9c-zr2b -c static-client -- curl -X POST http://localhost:19000/logging\?paths\=upstream:debug,http:debug,router:debug,config:debug

Make sure to append the correct component at the end of the curl command, i.e. logging?<component>=debug.

  • If curl is not able to be used in your pod, you can alternatively use kubectl port-forward <pod-name> 19000 to make the Envoy admin accessible. From another terminal window, you can then curl to change the log levels. The output you receive in the terminal will show the modified component log levels.

    curl -X POST http://localhost:19000/logging\?<component>\=debug

or for multiple components

    curl -X POST http://localhost:19000/logging\?paths\=<component>:debug,<component>=debug         

 

 

Access Envoy logs in Kubernetes

Accessing Envoy logs via pods can be done with the following command:

kubectl logs --follow pod/<pod-name> -c envoy-sidecar
  • The --follow flag provides a real time observation into Envoy logs. 

 

Setting and Accessing Envoy logs when not using Helm.

The following command will start an envoy side car proxy, set the log level to debug with -l debug and capture Envoy logs in envoy_logs.txt. The .txt file will need to be created before executing this command.

 

consul connect envoy -sidecar-for counting-1 -- -l debug --log-path envoy_logs.txt

To have granular control over the Envoy components that is needed to be debugged, use the following command:

 

consul connect envoy -sidecar-for counting-1 -- --log-path envoy_logs.txt --component-log-level upstream:debug,http:debug,router:debug,config:debug

Articles in this section

See more

Related articles