Expected Outcome
Envoy logs will be accessible and ready for debugging.
Primer
- Envoy: A service proxy for communication between applications within a service mesh.
- upstream: the upstream cluster in Envoy that will be receiving requests. The envoy logs will show any request/connect timeouts errors with upstream in debug mode
- http: any http request made through Envoy will be recorded here, including http requests to sidecar proxies or the Envoy admin UI.
- config: the config component is meant to collect any configuration updates to Envoy clusters(dynamic listeners, routers, clusters, and endpoints). Setting this in debug mode will reflect these changes.
- router: the router takes an incoming HTTP request, matches it to an upstream cluster, acquires a connection pool to a host in the upstream cluster, and forwards the request.
Use Case
There are times when services registered in Consul are not reachable, and observing logs for those services are invaluable in that moment. When connect is enabled, services have a sidecar proxy that leverages the Envoy proxy by default. Since service-to-service communication occurs through Envoy sidecar proxies, Envoy logs can shed light on errors between services. There are a few ways to access Envoy logs and set log levels to start debugging. In this article, we will be going over this with and without the Consul Helm Chart.
Procedure
Setting Envoy logs in the Helm configuration
The Consul helm chart uses envoyExtraArgs: to leverage Envoy command line options. One of the helpful options is --component-log-level. This provides granular control over setting log levels for Envoy components. In the example below, the components upstream, http, router and config are set to the debug log level. These four components are vital when debugging issues with requests between your services(sidecar proxies).
connectInject:
enabled: true
envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"
If you haven't set envoyExtraArgs: in consul-values.yaml just yet, you can set the log levels on the fly by using the following kubectl command:
kubectl exec pod/<pod-name> -c <container-name> -- curl -X POST http://localhost:19000/logging?config=debug
kubectl exec pod/static-client-5bf4575d9c-zr2b -c static-client -- curl -X POST http://localhost:19000/logging?config=debug
You can execute the kubectl command for each component or execute for multiple in one command:
kubectl exec pod/static-client-5bf4575d9c-zr2b -c static-client -- curl -X POST http://localhost:19000/logging\?paths\=upstream:debug,http:debug,router:debug,config:debug
Make sure to append the correct component at the end of the curl command, i.e. logging?<component>=debug.
- If curl is not able to be used in your pod, you can alternatively use
kubectl port-forward <pod-name> 19000to make the Envoy admin accessible. From another terminal window, you can then curl to change the log levels. The output you receive in the terminal will show the modified component log levels.
curl -X POST http://localhost:19000/logging\?<component>\=debug
or for multiple components
curl -X POST http://localhost:19000/logging\?paths\=<component>:debug,<component>=debug
Access Envoy logs in Kubernetes
Accessing Envoy logs via pods can be done with the following command:
kubectl logs --follow pod/<pod-name> -c envoy-sidecar
- The
--followflag provides a real time observation into Envoy logs.
Setting and Accessing Envoy logs when not using Helm.
The following command will start an envoy side car proxy, set the log level to debug with -l debug and capture Envoy logs in envoy_logs.txt. The .txt file will need to be created before executing this command.
consul connect envoy -sidecar-for counting-1 -- -l debug --log-path envoy_logs.txt
To have granular control over the Envoy components that is needed to be debugged, use the following command:
consul connect envoy -sidecar-for counting-1 -- --log-path envoy_logs.txt --component-log-level upstream:debug,http:debug,router:debug,config:debug