HashiCorp Diagnostics (hcdiag) – HashiCorp Help Center

HashiCorp Diagnostics hcdiag is a universal troubleshooting data gathering tool you can use to collect and archive important data from HashiCorp product installations. The tool can be used with Vault, Consul, Nomad and Terraform Enterprise. The information gathered by hcdiag is good for incident response and troubleshooting errors.

The output of these commands is stored in a temporary directory and bundled into a single archive as an output. To view all possible commands use -dryrun.

Prerequisites

Version(s)	Notes
All	hcdiag needs to be run on the server running the product(s).
0.3.0	TLS API limitations are resolved
0.2.0 and older	Limited TLS API support

The following subsections cover product specific prerequisite items such as environment variables required to successfully communicate with a given environment.

hcdiag Release

Determine the latest hcdiag binary version from https://releases.hashicorp.com/hcdiag/ and ensure your method of retrieval (i.e., curl or wget) takes this version into account. This ensures the provided diagnostics provide the highest degree of detail to be used by HashiCorp Support Engineering for analysis.

Consul

Required Environment Variables

CONSUL_HTTP_ADDR set to the address to the local Consul agent
CONSUL_TOKEN must be set to the API access token if ACLs are enabled

Tutorial

Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/

- https://learn.hashicorp.com/tutorials/consul/hcdiag-with-consul

Nomad

Note: The latest Nomad binary must be available in the same directory as hcdiag

Required Environment Variables

NOMAD_ADDR set to the address of the Nomad server
NOMAD_TOKEN must be set if ACLs are enabled
- Must have sufficient permissions. For more information see: https://www.vaultproject.io/docs/commands/debug

Tutorial

Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/

- https://learn.hashicorp.com/tutorials/nomad/hcdiag-with-nomad

Vault

Required Environment Variables

VAULT_ADDR set to the address of the Vault server
VAULT_TOKEN must be set to the Vault authentication token
- Alternatively, a token may also exist at ~/.vault_token
- If both are present, VAULT_TOKEN will be used
- Must have sufficient permissions. For more information see: https://www.vaultproject.io/docs/commands/debug

Tutorial

Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/

https://learn.hashicorp.com/tutorials/vault/hcdiag-with-vault

Terraform Enterprise

Required Environment Variables

TFE_HTTP_ADDR set to the url of the TFE server
- ```
export TFE_HTTP_ADDR = https://<hostnameFQDN>
```
TFE_TOKEN should be a generated Terraform Enterprise API token

Tutorial

Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/

- https://learn.hashicorp.com/tutorials/terraform/hcdiag-with-tfe

Submitting a ticket to Support

When submitting a ticket to Help Center, please submit hcdiag-{YYYY}-{MM}-{mm}T{nanos}Z.tar.gz. Please only submit the tar ball without renaming it instead of individual log files as Support Engineers would be able to diagnose the issue efficiently.

Usage

Input Flags


Argument	Description	Type	Default Value
dryrun	Perform a dry run to display commands without executing them	bool	false
os	Override operating system detection	string	"auto"
consul	Run Consul diagnostics	bool	false
nomad	Run Nomad diagnostics	bool	false
vault	Run Vault diagnostics	bool	false
includes	files or directories to include (comma-separated, file--globbing available if 'wrapped--in-single-quotes') e.g. '/var/log/consul- ,/var/log/nomad- '	string	""
include-since	Time range to include files, counting back from now. Takes a 'go-formatted' duration, usage examples: 72h, 25m, 45s, 120h1m90s	string	"72h"
destination	Path to the directory the bundle should be written in	string	"."
dest	Shorthand for -destination	string	"."
config	Path to HCL configuration file	string	""
debug-duration	Duration to run product debug bundle commands. Usage examples: 10s, 25m, 2h10m30s	string	"10s"
debug-interval	How long metrics collection intervals in product debug commands last. Usage examples: 10s, 25m, 2h10m30s	string	"5s"
serial	Run products in sequence rather than concurrently. Mostly for dev - use only if you want to be especially delicate with system load.	bool	false
version	Print the current version of hcdiag

Examples

Host diagnostics only

```
hcdiag 
```

Host and Vault diagnostics

```
hcdiag -vault 
```

Host, Consul, and Nomad diagnostics

```
hcdiag -consul -nomad 
```

Host and all available product diagnostics

```
hcdiag -all 
```

You can list the available flags available by running hcdiag --help as seen below:

$ hcdiag --help
Usage of hcdiag:
  -autodetect
           Auto-Detect installed products; any provided product flags will override this setting (default true)
  -config string
           Path to HCL configuration file
  -consul
           Run Consul diagnostics
  -debug-duration 00h00m00s
           How long to run product debug bundle commands. Provide a duration ex: 00h00m00s. See: -duration in `vault debug`, `consul debug`, and `nomad operator debug` (default 10s)
  -debug-interval 00h00m00s
           How long metrics collection intervals in product debug commands last. Provide a duration ex: 00h00m00s. See: -interval in `vault debug`, `consul debug`, and `nomad operator debug` (default 5s)
  -dest string
           Shorthand for -destination (default ".")
  -destination string
           Path to the directory the bundle should be written in (default ".")
  -dryrun
           Performing a dry run will display all commands without executing them
  -include-since 72h
           Alias for -since, will be overridden if -since is also provided, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s)
  -includes value
           files or directories to include (comma-separated, file-*-globbing available if 'wrapped-*-in-single-quotes')
e.g. '/var/log/consul-*,/var/log/nomad-*'
  -nomad
           Run Nomad diagnostics
  -os string
           Override operating system detection (default "auto")
  -serial
           Run products in sequence rather than concurrently
  -since 72h
           Collect information within this time. Takes a 'go-formatted' duration, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s)
  -terraform-ent
           (Experimental) Run Terraform Enterprise diagnostics
  -vault
           Run Vault diagnostics
  -version
           Print the current version of hcdiag

Installation

Download the binary with curl.

$ curl \
--silent \
--remote-name \
https://releases.hashicorp.com/hcdiag/0.4.0/hcdiag_0.4.0_linux_amd64.zip

Unzip and remove the archive.

$ unzip hcdiag_0.4.0_linux_amd64.zip && rm -f hcdiag_0.4.0_linux_amd64.zip

Finally add the hcdiag executable directory to the PATH so that you can invoke the command using just its name.

$ export PATH=$PWD:$PATH

Here is an example of using hcdiag with Vault

$ hcdiag -vault 
2022-02-24T14:50:46.821Z [INFO] hcdiag: Checking product availability 
2022-02-24T14:50:46.946Z [INFO] hcdiag: Gathering diagnostics 
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=host 
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker=stats 
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=vault 
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker="vault version" 
2022-02-24T14:50:46.991Z [INFO] hcdiag: running: seeker="vault status -format=json" 
2022-02-24T14:50:47.042Z [INFO] hcdiag: running: seeker="vault read sys/health -format=json" 
2022-02-24T14:50:47.088Z [INFO] hcdiag: running: seeker="vault read sys/seal-status -format=json" 
2022-02-24T14:50:47.138Z [INFO] hcdiag: running: seeker="vault read sys/host-info -format=json" 
2022-02-24T14:50:47.187Z [INFO] hcdiag: running: seeker="vault debug -output=hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz -duration=10s" 
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Results.json file: dest=hcdiag-2022-02-24T145046Z/Results.json 
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Manifest.json file: dest=hcdiag-2022-02-24T145046Z/Manifest.json 
2022-02-24T14:50:58.282Z [INFO] hcdiag: Compressed and archived output file: dest=./hcdiag-2022-02-24T145046Z.tar.gz

List the directory for tar+gzip archive files to discover the file that hcdiag created.

$ ls -l *.gz
-rw-r--r-- 1 vault vault 139143 Aug 10 21:18 hcdiag-2022-02-15T175841Z.tar.gz

The filename contains a hcdiag prefix, followed by a timestamp. You can unpack the archive and further examine its contents.

$ tar zxvf hcdiag-2022-02-24T145046Z.tar.gz 
hcdiag-2022-02-24T145046Z/Manifest.json 
hcdiag-2022-02-24T145046Z/Results.json 
hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz

The archive extracts three files into a temporary directory.

Manifest.json contains information describing the hcdiag run, including configuration options used, run duration, and a count of any errors encountered.
Results.json contains information about the environment and the output from an invocation of vault debug.
VaultDebug.tar.gz contains the output from invoking vault debug as described in

You can further explore the VaultDebug.tar.gz archive by unpacking it and examining its contents.

$ tar zxvf temp005038472/VaultDebug.tar.gz
VaultDebug/
VaultDebug/2021-08-10T21-18-42Z/
VaultDebug/2021-08-10T21-18-42Z/allocs.prof
VaultDebug/2021-08-10T21-18-42Z/block.prof
VaultDebug/2021-08-10T21-18-42Z/goroutine.prof
VaultDebug/2021-08-10T21-18-42Z/goroutines.txt
VaultDebug/2021-08-10T21-18-42Z/heap.prof
VaultDebug/2021-08-10T21-18-42Z/mutex.prof
VaultDebug/2021-08-10T21-18-42Z/profile.prof
VaultDebug/2021-08-10T21-18-42Z/threadcreate.prof
VaultDebug/2021-08-10T21-18-42Z/trace.out
VaultDebug/2021-08-10T21-18-52Z/
VaultDebug/2021-08-10T21-18-52Z/allocs.prof
VaultDebug/2021-08-10T21-18-52Z/block.prof
VaultDebug/2021-08-10T21-18-52Z/goroutine.prof
VaultDebug/2021-08-10T21-18-52Z/goroutines.txt
VaultDebug/2021-08-10T21-18-52Z/heap.prof
VaultDebug/2021-08-10T21-18-52Z/mutex.prof
VaultDebug/2021-08-10T21-18-52Z/threadcreate.prof
VaultDebug/config.json
VaultDebug/host_info.json
VaultDebug/index.json
VaultDebug/metrics.json
VaultDebug/replication_status.json
VaultDebug/server_status.json
VaultDebug/vault.log

If you are using a token with insufficient privileges, VaultDebug bundle gets generated but most of the data would be missing. Refer to the VaultDebug/index.json file for errors related to Permission Denied for certain paths when there is a token missing or if the token does not have sufficient privileges. Data from unauthenticated points still gets captured in the Vault Debug bundle without a token being passed

Example, to include only 24 hours of log output:

$ hcdiag -vault -debug-duration 24h

Troubleshooting

Issue: Error reading logs from the journal

Example:

[ERROR] hcdiag: journalctl -u vault: message="unable to read logs, is your user allowed? try sudo?"
output=
| Hint: You are currently not seeing messages from other users and the system.
| Users in groups 'adm', 'systemd-journal' can see all messages.
| Pass -q to turn off this notice.
| -- No entries --

Solution: Re-run hcdiag as a user with permissions to read the journal

References

For more information about using hcdiag, please refer to the below documentation:

https://learn.hashicorp.com/tutorials/terraform/hcdiag-with-tfe

https://learn.hashicorp.com/tutorials/vault/hcdiag-with-vault

https://learn.hashicorp.com/tutorials/nomad/hcdiag-with-nomad

https://learn.hashicorp.com/tutorials/consul/hcdiag-with-consul

Prerequisites

hcdiag Release

Consul

Required Environment Variables

Tutorial

Nomad

Required Environment Variables

Tutorial

Vault

Required Environment Variables

Tutorial

Terraform Enterprise

Required Environment Variables

Tutorial

Submitting a ticket to Support

Usage

Input Flags

Examples

Installation

Troubleshooting

References

Articles in this section

Related articles