Introduction

This article provides a comprehensive guide on collecting performance profiles from a running Consul agent using Go's pprof profiling tools. These profiles are invaluable for diagnosing and resolving common performance bottlenecks such as high CPU utilization, goroutine leaks, and excessive heap memory consumption. We will cover the preferred method using the consul debug command, and also demonstrate manual collection via curl commands or the consul-pprof.sh helper script for situations where consul debug might not be immediately feasible.

Expected Outcome

By following this guide, you will be able to:

• Understand the different pprof endpoints exposed by Consul and their purpose.
• Manually collect various pprof profiles (Heap, CPU, Trace, Goroutine) from a Consul agent using curl.
• Automate pprof collection using the consul-pprof.sh script for efficient data gathering.
• Utilize the consul debug command to capture a comprehensive support bundle, including pprof data and debug logs, which is the preferred method for complete diagnostics.
• Prepare collected pprof files for analysis with go tool pprof or for submission to HashiCorp Support.

Prerequisites

To successfully collect pprof profiles, ensure you have the following:

• Access to a running Consul agent: The agent should be accessible via its HTTP API.
• consul CLI tool: For using the consul debug command.
• curl utility: For manual collection.
• go runtime and go tool pprof: While not strictly required for collection, these are essential for analyzing the profiles.
• Consul ACL Token (if ACLs are enabled): A management or operator token with sufficient permissions to access the /debug/pprof endpoints and execute consul debug.
• Network connectivity: Ensure no firewalls or proxies are blocking access to the Consul agent's debug endpoints.
• Bash environment: If using the consul-pprof.sh script.

Use Case

Collecting pprof profiles is essential for several scenarios, including:

• Troubleshooting high CPU usage: Identify hot spots in the Consul codebase consuming excessive CPU cycles.
• Diagnosing memory leaks: Pinpoint memory accumulation issues by analyzing heap profiles.
• Detecting goroutine leaks: Identify goroutines that are unexpectedly blocked or not terminating, leading to resource exhaustion.
• Performance optimization: Gain insights into the runtime behavior of your Consul agents to proactively optimize configurations and workload patterns.
• Providing data for support cases: When experiencing performance issues, HashiCorp Support often requests these profiles for in-depth analysis. Using consul debug is preferred as it provides a more complete picture, including relevant logs.

Procedure

Consul exposes several pprof endpoints, each providing a distinct type of performance data. The table below provides an overview:

Preferred Method: Using consul debug

The preferred method to capture pprof data and other valuable troubleshooting information is by using the consul debug command. This command collects a comprehensive support bundle, which includes pprof profiles, agent configuration, relevant logs, and more, providing a holistic view for diagnosing issues.

Important Prerequisite for consul debug pprof collection:

For consul debug to successfully collect pprof data, the enable_debug configuration parameter in your Consul agent's configuration file must be set to true.

• Configuration Example (within your Consul agent config file):

  enable_debug = true

  • Restart Required: If enable_debug is not currently set to true, you will need to restart your Consul agent for this change to take effect.

Reason for Preference:

The consul debug command is preferred because it gathers a complete set of diagnostic information that is invaluable for troubleshooting, including:

• Debug Logs: Crucial for understanding the agent's behavior leading up to an issue.
• Configuration: Provides insight into the agent's current settings.
• Internal State: Offers a deeper look into Consul's operational state.

Executing consul debug:

Once enable_debug is set to true and the agent is running, execute the consul debug command. You may need to specify the Consul HTTP address and token.

This command will typically create a gzipped tarball (e.g., consul-debug-2025-06-17T15-30-00Z.tar.gz) containing all the collected data, including the pprof profiles within a dedicated directory inside the bundle.

Alternative Methods: Manual Collection via curl or consul-pprof.sh

There might be situations where setting enable_debug to true and restarting the Consul agent is not feasible. For instance, if an issue is actively occurring and restarting the agent would temporarily mitigate the problem, making it unpredictable when the issue would resurface. In such cases, manually collecting pprof data without a restart can be extremely helpful.

Manual Collection via curl

For direct control over the profiling process, you can manually collect profiles using curl. This method requires direct network reachability to the Consul agent and proper authentication if ACLs are enabled.

1. Set Environment Variables: It's recommended to set environment variables for the Consul HTTP address and ACL token for easier execution. Replace <your-management-token> with your actual token.

   export CONSUL_HTTP_ADDR=http://localhost:8500
   export CONSUL_HTTP_TOKEN=<your-management-token>
   

2. Execute curl Commands: Run the following curl commands to collect the different profile types. The -s flag silences progress output, -H adds the ACL token header, and -o specifies the output filename.
   • Heap Profile (Memory Usage Snapshot):

     curl -s "${CONSUL_HTTP_ADDR}/debug/pprof/heap" \
       -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" \
       -o heap.prof

   • CPU Profile (e.g., 30-second sample):

     curl -s "${CONSUL_HTTP_ADDR}/debug/pprof/profile?seconds=30" \
       -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" \
       -o cpu.prof

   • Trace Profile (Execution Trace with Goroutine Activity):

     curl -s "${CONSUL_HTTP_ADDR}/debug/pprof/trace?seconds=30" \
       -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" \
       -o trace.out

   • Goroutine Profile (Current Stack Trace Summary):

     curl -s "${CONSUL_HTTP_ADDR}/debug/pprof/goroutine" \
       -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" \
       -o goroutine.prof

   • Goroutine (Debug=2) Profile (Detailed Stack Traces and States):

     curl -s "${CONSUL_HTTP_ADDR}/debug/pprof/goroutine?debug=2" \
       -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" \
       -o goroutine-raw.prof

Once collected, these .prof and .out files can be analyzed using the go tool pprof command or attached to support tickets.

Automated Collection Using consul-pprof.sh

For a more streamlined and automated approach, especially when collecting multiple profiles or on remote systems, the consul-pprof.sh helper script is highly recommended when consul debug is not viable. This script automates several steps:

• Validation: Checks agent availability and ACL token access.
• Comprehensive Collection: Gathers all five pprof profile types.
• Output Validation: Verifies the integrity of each collected profile.
• Organized Storage: Saves all profiles in a timestamped directory under /tmp (or a specified output directory).

Usage:

Download the consul-pprof.sh script (often available as a Gist from HashiCorp support or community resources) and make it executable:

chmod +x consul-pprof.sh
./consul-pprof.sh [OPTIONS]

Common Options:

Example:

export CONSUL_HTTP_ADDR="http://10.0.0.12:8500"
export CONSUL_HTTP_TOKEN="root"

./consul-pprof.sh \
    --duration 60 \
    --output-dir /tmp/my-consul-pprof

The profiles will be saved in the specified directory (e.g., /tmp/my-consul-pprof/consul-pprof-2025-06-17_15-00-00/) and automatically validated.

ACL and Debug Mode Notes for consul-pprof.sh:

• The consul-pprof.sh script is designed to work regardless of the enable_debug setting on the Consul agent, as it directly queries the pprof HTTP endpoints.
• It primarily uses the X-Consul-Token header for authentication, allowing it to bypass some ACL restrictions if the provided token has sufficient privileges.
• For the script to capture full stack traces and verbose output, ensure that the debug endpoints are not restricted by network firewalls or Layer 7 proxies.

Additional Information

• go tool pprof: This is the primary command-line tool for visualizing and analyzing the generated pprof files. It can render flame graphs, call graphs, and more.
  • Official go tool pprof documentation
  • go tool pprof usage
• Consul debug CLI command: This is the recommended command for comprehensive data collection. It gathers pprof data, logs, configuration, and internal state.
  • Official Consul debug CLI documentation
• Consul enable_debug Configuration: This agent configuration parameter controls whether additional debug endpoints, including pprof endpoints accessible by consul debug, are enabled.
  • Official enable_debug documentation
• Consul HTTP API: debug.go: The source code where Consul registers its pprof debug endpoints.
• Go net/http/pprof package: The standard Go library package that provides the pprof HTTP handlers that Consul integrates.
• consul-pprof.sh script (Gist): Ensure you have the latest version of the consul-pprof.sh script. (It would be beneficial to link directly to a stable Gist or repository if available).

Need Help?

When opening a support ticket with HashiCorp for performance troubleshooting, please attach the generated consul debug bundle (e.g., consul-debug-*.tar.gz). If a consul debug bundle was not feasible to collect, please provide the manually generated .prof and .out files. These profiles are crucial for our engineers to effectively diagnose and assist with your Consul performance issues. Providing a detailed description of the problem and the steps leading to it will also greatly aid in the resolution process.