HashiCorp Diagnostics (hcdiag)

Avatar
Zaid Baban
  • Updated
Follow
HashiCorp Diagnostics hcdiag is a universal troubleshooting data gathering tool you can use to collect and archive important data from HashiCorp product installations. The tool can be used with Vault, Consul and Nomad. The information gathered by hcdiag is good for incident response and troubleshooting errors.
The output of these commands is stored in a temporary directory and bundled into a single archive as an output. To view all possible commands use -dryrun.

Prerequisites

*Note* hcdiag needs to be run on the server running the product(s), and currently has limited TLS API support.
The following subsections cover product specific prerequisite items such as environment variables required to successfully communicate with a given environment.

Consul

Required Environment Variables

Tutorial

Nomad

Note: The latest Nomad binary must be available in the same directory as hcdiag

Required Environment Variables

Tutorial

Vault

Required Environment Variables

Tutorial

 

Submitting a ticket to Support

When submitting a ticket to Help Center, please submit hcdiag-{YYYY}-{MM}-{mm}T{nanos}Z.tar.gz. Please only submit the tar ball without renaming it instead of individual log files as Support Engineers would be able to diagnose the issue efficiently.

 

Usage

Input Flags

Argument
Description
Type
Default Value
 dryrun
 Perform a dry run to display commands without executing them
 bool
 false
 os
 Override operating system detection
 string
 "auto"
 consul
 Run Consul diagnostics
 bool
 false
 nomad
 Run Nomad diagnostics
 bool
 false
 vault
 Run Vault diagnostics
 bool
 false
 includes
 files or directories to include (comma-separated, file--globbing available if 'wrapped--in-single-quotes') e.g. '/var/log/consul- ,/var/log/nomad- '
 string
 ""
 include-since
 Time range to include files, counting back from now. Takes a 'go-formatted' duration, usage examples: 72h, 25m, 45s, 120h1m90s
 string
 "72h"
 destination
 Path to the directory the bundle should be written in
 string
 "."
 dest
 Shorthand for -destination
 string
 "."
 config
 Path to HCL configuration file
 string
 ""
 serial
 Run products in sequence rather than concurrently. Mostly for dev - use only if you want to be  especially delicate with system load.
 bool
 false
 version
 Print the current version of hcdiag
 
 
 

 

Examples

  • Host diagnostics only
    • hcdiag
  • Host and Vault diagnostics
    • hcdiag -vault
  • Host, Consul, and Nomad diagnostics
    • hcdiag -consul -nomad
  • Host and all available product diagnostics
    • hcdiag -all
You can list the available flags available by running hcdiag --help as seen below: 
$ hcdiag --help
Usage of hcdiag:
-all
    DEPRECATED: Run all available product diagnostics
-config string
    Path to HCL configuration file
-consul
    Run Consul diagnostics
-dest string
    Shorthand for -destination (default ".")
-destination string
    Path to the directory the bundle should be written in (default ".")
-dryrun
    Performing a dry run will display all commands without executing them
-include-since 72h
    Alias for -since, will be overridden if -since is also provided, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s)
-includes value
    files or directories to include (comma-separated, file-*-globbing available if 'wrapped-*-in-single-quotes')
    e.g. '/var/log/consul-*,/var/log/nomad-*'
-nomad
    Run Nomad diagnostics
-os string
    Override operating system detection (default "auto")
-serial
    Run products in sequence rather than concurrently
-since 72h
    Collect information within this time. Takes a 'go-formatted' duration, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s)
-terraform-ent
    (Experimental) Run Terraform Enterprise diagnostics
-vault
    Run Vault diagnostics
-version
    Print the current version of hcdiag

Installation

Download the binary with curl.
$ curl \
--silent \
--remote-name \
https://releases.hashicorp.com/hcdiag/0.2.0/hcdiag_0.2.0_linux_amd64.zip
Unzip and remove the archive.
$ unzip hcdiag_0.2.0_linux_amd64.zip && rm -f hcdiag_0.2.0_linux_amd64.zip
Finally add the hcdiag executable directory to the PATH so that you can invoke the command using just its name.
$ export PATH=$PWD:$PATH
Here is an example of using hcdiag with Vault
$ hcdiag -vault 
2022-02-24T14:50:46.821Z [INFO] hcdiag: Checking product availability 
2022-02-24T14:50:46.946Z [INFO] hcdiag: Gathering diagnostics 
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=host 
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker=stats 
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=vault 
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker="vault version" 
2022-02-24T14:50:46.991Z [INFO] hcdiag: running: seeker="vault status -format=json" 
2022-02-24T14:50:47.042Z [INFO] hcdiag: running: seeker="vault read sys/health -format=json" 
2022-02-24T14:50:47.088Z [INFO] hcdiag: running: seeker="vault read sys/seal-status -format=json" 
2022-02-24T14:50:47.138Z [INFO] hcdiag: running: seeker="vault read sys/host-info -format=json" 
2022-02-24T14:50:47.187Z [INFO] hcdiag: running: seeker="vault debug -output=hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz -duration=10s" 
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Results.json file: dest=hcdiag-2022-02-24T145046Z/Results.json 
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Manifest.json file: dest=hcdiag-2022-02-24T145046Z/Manifest.json 
2022-02-24T14:50:58.282Z [INFO] hcdiag: Compressed and archived output file: dest=./hcdiag-2022-02-24T145046Z.tar.gz
List the directory for tar+gzip archive files to discover the file that hcdiag created.
$ ls -l *.gz
-rw-r--r-- 1 vault vault 139143 Aug 10 21:18 hcdiag-2022-02-15T175841Z.tar.gz
The filename contains a hcdiag  prefix, followed by a timestamp. You can unpack the archive and further examine its contents.
$ tar zxvf hcdiag-2022-02-24T145046Z.tar.gz 
hcdiag-2022-02-24T145046Z/Manifest.json 
hcdiag-2022-02-24T145046Z/Results.json 
hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz
The archive extracts three files into a temporary directory.
  • Manifest.json contains information describing the hcdiag run, including configuration options used, run duration, and a count of any errors encountered.
  • Results.json contains information about the environment and the output from an invocation of vault debug.
  • VaultDebug.tar.gz contains the output from invoking vault debug as described in
You can further explore the VaultDebug.tar.gz archive by unpacking it and examining its contents.
$ tar zxvf temp005038472/VaultDebug.tar.gz
VaultDebug/
VaultDebug/2021-08-10T21-18-42Z/
VaultDebug/2021-08-10T21-18-42Z/allocs.prof
VaultDebug/2021-08-10T21-18-42Z/block.prof
VaultDebug/2021-08-10T21-18-42Z/goroutine.prof
VaultDebug/2021-08-10T21-18-42Z/goroutines.txt
VaultDebug/2021-08-10T21-18-42Z/heap.prof
VaultDebug/2021-08-10T21-18-42Z/mutex.prof
VaultDebug/2021-08-10T21-18-42Z/profile.prof
VaultDebug/2021-08-10T21-18-42Z/threadcreate.prof
VaultDebug/2021-08-10T21-18-42Z/trace.out
VaultDebug/2021-08-10T21-18-52Z/
VaultDebug/2021-08-10T21-18-52Z/allocs.prof
VaultDebug/2021-08-10T21-18-52Z/block.prof
VaultDebug/2021-08-10T21-18-52Z/goroutine.prof
VaultDebug/2021-08-10T21-18-52Z/goroutines.txt
VaultDebug/2021-08-10T21-18-52Z/heap.prof
VaultDebug/2021-08-10T21-18-52Z/mutex.prof
VaultDebug/2021-08-10T21-18-52Z/threadcreate.prof
VaultDebug/config.json
VaultDebug/host_info.json
VaultDebug/index.json
VaultDebug/metrics.json
VaultDebug/replication_status.json
VaultDebug/server_status.json
VaultDebug/vault.log
Example, to include only 24 hours of log output:
$ hcdiag -vault -since 24h

 

Troubleshooting

Issue: Error reading logs from the journal

Example:

[ERROR] hcdiag: journalctl -u vault: message="unable to read logs, is your user allowed? try sudo?"
output=
| Hint: You are currently not seeing messages from other users and the system.
| Users in groups 'adm', 'systemd-journal' can see all messages.
| Pass -q to turn off this notice.
| -- No entries --
Solution: Re-run hcdiag as a user with permissions to read the journal

References

For more information about using hcdiag, please refer to the below documentation:

Articles in this section

Related articles