How-to Enhance Consul and Nomad Logging

Avatar
Mikael Sikora
  • Updated

Introduction

Consul and Nomad are two powerful open-source tools that can be used to manage distributed systems. However, when problems arise, it can be difficult to troubleshoot them if you don't have good logging in place. This article will discuss ways to enhance Consul and Nomad logging, which will help you to troubleshoot and debug your systems more effectively.

We will cover the following topics in this article:

  • Capturing the activity of Consul and Nomad
  • Troubleshooting Consul and Nomad
  • Putting logging in the Nomad and Consul configuration files
  • Issuing CLI and API commands from a host outside of the clusters (except for journald)
  • Other tips for enhancing Consul and Nomad logging

Expected Outcome

  • Enable detailed trace logging for both Consul and Nomad, giving you deeper insights into their operations.
  • Capture Consul and Nomad activity for troubleshooting without modifying your production configuration, using convenient CLI commands or API calls.

Prerequisites

  • Consul and Nomad binaries for CLI commands.
  • Connectivity, authentication, and permissions to access the Consul and Nomad services.  This will include ACL tokens (at least management level) if Consul and/or Nomad ACL are activated.
  • The Kubernetes command-line tool, kubectl.

Use Case

  • Debugging unexpected behavior: If Consul or Nomad is behaving unexpectedly, such as failing to register services, experiencing network connectivity problems, or showing performance degradation, enhanced logging can provide insights into the root cause.
  • Investigating performance issues: When Consul or Nomad is running slower than expected, detailed logs can help identify bottlenecks or resource constraints.
  • Troubleshooting failed deployments: If deployments in Nomad are failing, logs can help pinpoint the stage where the failure occurs and provide clues about why it's happening.
  • Auditing system activity: Enhanced logging can be used to track changes made to Consul or Nomad configurations, deployments, and service registrations, providing an audit trail for security and compliance purposes.
  • Monitoring system health: Logs can be integrated with monitoring tools to provide real-time alerts on critical events or errors, enabling proactive system management.

Procedure

The following examples provide guidance on setting up logging for your Consul and Nomad servers and clients, including how to capture streaming output. While configuring logging directly within your Nomad and Consul configuration files is the recommended approach, this guide also offers alternative methods using CLI commands and API calls that can be issued remotely (with the exception of journald).

For your convenience, references to the relevant documentation for each command are included at the bottom of this page.

Consul

  • Consul Server/Client configuration files.
  • Create directory /opt/consul/logs.
  • Provide the new directory with appropriate permissions and ownership.
log_level            = "TRACE"
log_file             = "/opt/consul/logs/"
log_rotate_duration  = "8h"
log_rotate_max_files = 3

Consul CLI

  • Consul Agent Monitor:  This live streams the agent output and stores it in a file.
# SET VARIABLES

consul_token=xxxxxxxxxxxxxx
host_addr=<IP or Hostname>

# TEST CONNECTIVITY

consul members -token=${consul_token} -http-addr=http://${host_addr}:8500

# STREAMING

consul monitor -log-level=TRACE -token=${consul_token} -http-addr=http://${host_addr}:8500 > ${host_addr}_consul_monitor.log

Consul API

  • Consul Agent Monitor:  This live streams the agent output and stores it in a file.
# TEST CONNECTIVITY

curl --header "X-Consul-Token: ${consul_token}" http://${host_addr}:8500/v1/agent/members?loglevel=trace

# STREAMING

curl --header "X-Consul-Token: ${consul_token}" http://$host_addr:8500/v1/agent/monitor?loglevel=trace > ${host_addr}_consul_monitor.log

Bash

  • Journald:  Get all data for Consul from start to present and stores it in a file.
# ONE TIME CAPTURE

sudo journalctl --boot -x -p debug -u consul --no-pager > $(hostname -I | awk '{print $1}')_journalctl_consul.log

Docker

Bash

  • Journald:  Get all data for Docker from start to present and stores it in a file.
# ONE TIME CAPTURE

sudo journalctl --boot -x -p debug -u docker --no-pager >${host_addr}_journalctl_docker.log

Kubernetes

Get Names of NAMESPACES, PODS, and CONTAINERS

  • kubectl get:  Get all the namespaces, pods, and container names.  This is needed to obtain log output.
echo -e "NAMESPACE\tPOD\tCONTAINERS" && kubectl get pods --all-namespaces --output jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .spec.containers[*]}{.name}{"\t"}{end}{"\n"}{end}'
  • kubectl get: Get the container names for pod frontend-5dfb4f6594-szwdq.
kubectl get pod frontend-5dfb4f6594-szwdq --namespace default --output jsonpath="{.spec.containers[*].name}"

Get Logs

  • kubectl logs:  Get the logs for pods and containers.
# Syntax:  kubectl logs <pod-name> --namespace <namespace-name> --container <container-name> 

# ONE TIME CAPTURE

# Output for pod frontend-5dfb4f6594-szwdq, all container logs:
kubectl logs frontend-5dfb4f6594-szwdq --namespace default --all-containers=true

# Output for pod frontend-5dfb4f6594-szwdq, container consul-dataplane:
kubectl logs frontend-5dfb4f6594-szwdq --namespace default --container consul-dataplane

# STREAMING

# Stream output for pod frontend-5dfb4f6594-szwdq, container frontend
kubectl logs frontend-5dfb4f6594-szwdq --namespace default --container frontend --follow

Get Events

  • kubectl events:  Get the event occurrences, that are typically generated for significant changes or issues, such as pod creation, deletion, failures, or status changes.
# ONE TIME

# Output all the events
kubectl events --all-namespaces

# STREAMING

# Stream output for namespace consul
kubectl events --namespace consul --watch

Nomad

  • Nomad Server/Client configuration files:
    • Create directory /opt/nomad/logs.
    • Provide the new directory with appropriate permissions and ownership.
log_level            = "TRACE"
log_file             = "/opt/nomad/logs/"
log_rotate_duration  = "8h"
log_rotate_max_files = 3

Nomad CLI

  • Allocation Dump, One Time Capture:  This will dump what is currently in the allocation log for specific allocation and task and store it in a file.  Run these right after an issue occurs.
  • Nomad Agent Monitor, Streaming:  This live streams the agent output and stores it in a file.  This is what is captured in the log_file from the configuration settings above.  Run this continuously to capture an issue that is going to happen.
# SET INITIAL VARIABLES

host_addr=<IP or Hostname>
nomad_token=xxxxxxxxxxxxxxx  ##  If using ACL

# TEST CONNECTION

nomad server members -token=${nomad_token} -address=http://${host_addr}:4646
nomad node status -token=${nomad_token} -address=http://${host_addr}:4646

# SET ADDITIONAL VARIABLES

# Get job names
nomad status -token=${nomad_token} -address=http://${host_addr}:4646
job_name=<job-name>

# Get task and alloc IDs
nomad job inspect ${job_name} | jq -r '.Job.TaskGroups[]?.Tasks[]?.Name'  ## Display the task names
task_id=<task name>
nomad job status ${job_name}  ## display the alloc-id(s)
alloc_id=<alloc-id>

# ONE TIME CAPTURE

# Allocation stderr
nomad alloc logs -token=${nomad_token} -address=http://${host_addr}:4646 -task ${task_id} -stderr -verbose ${alloc_id} > ${host_addr}_nomad_alloc_stderr.log

# Allocation stdout
nomad alloc logs -token=${nomad_token} -address=http://${host_addr}:4646 -task ${task_id} -verbose ${alloc_id} > ${host_addr}_nomad_alloc_stdout.log

# STREAMING

nomad monitor -log-level=TRACE -token=${nomad_token} -address=http://${host_addr}:4646 > ${host_addr}_nomad_monitor.log

Nomad API

  • Eventstream:  This is a live stream of Nomad deployments, evaluations, and allocations as they happen and stores it in a file.
# STREAMING

curl -s -v -N --header "X-Nomad-Token: <Nomad-Token>" http://${host_addr}:4646/v1/event/stream > "${host_addr}_eventstream.json" 2> "${host_addr}_eventstream.err"

Bash

  • journald:  Get all data for Nomad from start to present and stores it in a file.
# ONE TIME CAPTURE

sudo journalctl --boot -x -p debug -u nomad --no-pager > $(hostname -I | awk '{print $1}')_journalctl_nomad.log

Additional Information

Logging References

Metrics References

Articles in this section

See more

Related articles