How-to Enhance Consul and Nomad Logging

Avatar
Mikael Sikora
  • Updated

Introduction

Expected Outcome

Trace logging output of Consul and Nomad, with options of capturing the Consul and Nomad activity without making any changes to production by using CLI commands or API calls.

Prerequisites (if applicable)

  • Consul and Nomad binaries for CLI commands.
  • Connectivity, authentication, and permissions to access the Consul and Nomad services.  This will include ACL tokens (at least management level) if Consul and/or Nomad ACL are activated.

Use Case

Troubleshooting Consul and Nomad.

Procedure

Following are examples on setting up logging on your Nomad and Consul servers and clients, along with capturing streamed output.  Putting logging in the Nomad and Consul configuration files is ideal.  CLI and API commands are included, that can be issued from a host outside of the clusters (except for journald), as an alternative.  At the bottom are documentation references for each of the commands.

Consul

  • Consul Server/Client configuration files.
  • Create directory /opt/consul/logs.
  • Provide the new directory with appropriate permissions and ownership.
log_level            = "TRACE"
log_file             = "/opt/consul/logs/"
log_rotate_duration  = "8h"
log_rotate_max_files = 3

Consul CLI

  • Consul Agent Monitor:  This live streams the agent output and stores it in a file.
# SET VARIABLES

consul_token=xxxxxxxxxxxxxx
host_addr=<IP or Hostname>

# TEST CONNECTIVITY

consul members -token=${consul_token} -http-addr=http://${host_addr}:8500

# STREAMING

consul monitor -log-level=TRACE -token=${consul_token} -http-addr=http://${host_addr}:8500 > ${host_addr}_consul_monitor.log

Consul API

  • Consul Agent Monitor:  This live streams the agent output and stores it in a file.
# TEST CONNECTIVITY

curl --header "X-Consul-Token: ${consul_token}" http://${host_addr}:8500/v1/agent/members?loglevel=trace

# STREAMING

curl --header "X-Consul-Token: ${consul_token}" http://$host_addr:8500/v1/agent/monitor?loglevel=trace > ${host_addr}_consul_monitor.log

Bash

  • Journald:  Get all data for Consul from start to present and stores it in a file.
# ONE TIME CAPTURE

sudo journalctl --boot -x -p debug -u consul --no-pager > $(hostname -I | awk '{print $1}')_journalctl_consul.log

Docker

Bash

  • Journald:  Get all data for Docker from start to present and stores it in a file.
# ONE TIME CAPTURE

sudo journalctl --boot -x -p debug -u docker --no-pager >${host_addr}_journalctl_docker.log

Nomad

  • Nomad Server/Client configuration files:
    • Create directory /opt/nomad/logs.
    • Provide the new directory with appropriate permissions and ownership.
log_level            = "TRACE"
log_file             = "/opt/nomad/logs/"
log_rotate_duration  = "8h"
log_rotate_max_files = 3

Nomad CLI

  • Allocation Dump, One Time Capture:  This will dump what is currently in the allocation log for specific allocation and task and store it in a file.  Run these right after an issue occurs.
  • Nomad Agent Monitor, Streaming:  This live streams the agent output and stores it in a file.  This is what is captured in the log_file from the configuration settings above.  Run this continuously to capture an issue that is going to happen.
# SET INITIAL VARIABLES

host_addr=<IP or Hostname>
nomad_token=xxxxxxxxxxxxxxx  ##  If using ACL

# TEST CONNECTION

nomad server members -token=${nomad_token} -address=http://${host_addr}:4646
nomad node status -token=${nomad_token} -address=http://${host_addr}:4646

# SET ADDITIONAL VARIABLES

nomad status -token=${nomad_token} -address=http://${host_addr}:4646  ## Get job names
job_name=<job-name>
nomad job inspect ${job_name} | jq -r '.Job.TaskGroups[]?.Tasks[]?.Name'  ## Display the task names
task_id=<task name>
nomad job status ${job_name}  ## display the alloc-id(s)
alloc_id=<alloc-id>

# ONE TIME CAPTURE

nomad alloc logs -token=${nomad_token} -address=http://${host_addr}:4646 -task ${task_id} -stderr -verbose ${alloc_id} > ${host_addr}_nomad_alloc_stderr.log

nomad alloc logs -token=${nomad_token} -address=http://${host_addr}:4646 -task ${task_id} -verbose ${alloc_id} > ${host_addr}_nomad_alloc_stdout.log

# STREAMING

nomad monitor -log-level=TRACE -token=${nomad_token} -address=http://${host_addr}:4646 > ${host_addr}_nomad_monitor.log

Nomad API

  • Eventstream:  This is a live stream of Nomad deployments, evaluations, and allocations as they happen and stores it in a file.
# STREAMING

curl -s -v -N --header "X-Nomad-Token: <Nomad-Token>" http://${host_addr}:4646/v1/event/stream > "${host_addr}_eventstream.json" 2> "${host_addr}_eventstream.err"

Bash

  • journald:  Get all data for Nomad from start to present and stores it in a file.
# ONE TIME CAPTURE

sudo journalctl --boot -x -p debug -u nomad --no-pager > $(hostname -I | awk '{print $1}')_journalctl_nomad.log

Additional Information

Logging References

Metrics References

Articles in this section

See more

Related articles