HashiCorp Diagnostics
hcdiag
is a universal troubleshooting data gathering tool you can use to collect and archive important data from HashiCorp product installations. The tool can be used with Vault, Consul and Nomad. The information gathered by hcdiag
is good for incident response and troubleshooting errors.The output of these commands is stored in a temporary directory and bundled into a single archive as an output. To view all possible commands use
-dryrun
.Prerequisites
*Note* hcdiag needs to be run on the server running the product(s), and currently has limited TLS API support.
The following subsections cover product specific prerequisite items such as environment variables required to successfully communicate with a given environment.
Consul
Required Environment Variables
-
CONSUL_HTTP_ADDR set to the address to the local Consul agent
-
CONSUL_TOKEN must be set to the API access token if ACLs are enabled
Tutorial
Nomad
Note: The latest Nomad binary must be available in the same directory as
hcdiag
Required Environment Variables
-
NOMAD_ADDR set to the address of the Nomad server
-
NOMAD_TOKEN must be set if ACLs are enabled
- Must have sufficient permissions. For more information see: https://www.vaultproject.io/docs/commands/debug
Tutorial
Vault
Required Environment Variables
-
VAULT_ADDR set to the address of the Vault server
-
VAULT_TOKEN must be set to the Vault authentication token
-
Alternatively, a token may also exist at
~/.vault_token
-
If both are present, VAULT_TOKEN will be used
-
Must have sufficient permissions. For more information see: https://www.vaultproject.io/docs/commands/debug
-
Tutorial
Submitting a ticket to Support
When submitting a ticket to Help Center, please submit hcdiag-{YYYY}-{MM}-{mm}T{nanos}Z.tar.gz. Please only submit the tar ball without renaming it instead of individual log files as Support Engineers would be able to diagnose the issue efficiently.
Usage
Input Flags
Argument
|
Description
|
Type
|
Default Value
|
---|---|---|---|
dryrun
|
Perform a dry run to display commands without executing them
|
bool
|
false
|
os
|
Override operating system detection
|
string
|
"auto"
|
consul
|
Run Consul diagnostics
|
bool
|
false
|
nomad
|
Run Nomad diagnostics
|
bool
|
false
|
vault
|
Run Vault diagnostics
|
bool
|
false
|
includes
|
files or directories to include (comma-separated, file--globbing available if 'wrapped--in-single-quotes') e.g. '/var/log/consul- ,/var/log/nomad- '
|
string
|
""
|
include-since
|
Time range to include files, counting back from now. Takes a 'go-formatted' duration, usage examples: 72h, 25m, 45s, 120h1m90s
|
string
|
"72h"
|
destination
|
Path to the directory the bundle should be written in
|
string
|
"."
|
dest
|
Shorthand for -destination
|
string
|
"."
|
config
|
Path to HCL configuration file
|
string
|
""
|
serial
|
Run products in sequence rather than concurrently. Mostly for dev - use only if you want to be especially delicate with system load.
|
bool
|
false
|
version
|
Print the current version of hcdiag
|
|
|
Examples
-
Host diagnostics only
-
hcdiag
-
Host and Vault diagnostics
-
hcdiag -vault
-
Host, Consul, and Nomad diagnostics
-
hcdiag -consul -nomad
-
Host and all available product diagnostics
-
hcdiag -all
You can list the available flags available by running
hcdiag --help
as seen below: $ hcdiag --help
Usage of hcdiag:
-all
DEPRECATED: Run all available product diagnostics
-config string
Path to HCL configuration file
-consul
Run Consul diagnostics
-dest string
Shorthand for -destination (default ".")
-destination string
Path to the directory the bundle should be written in (default ".")
-dryrun
Performing a dry run will display all commands without executing them
-include-since 72h
Alias for -since, will be overridden if -since is also provided, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s)
-includes value
files or directories to include (comma-separated, file-*-globbing available if 'wrapped-*-in-single-quotes')
e.g. '/var/log/consul-*,/var/log/nomad-*'
-nomad
Run Nomad diagnostics
-os string
Override operating system detection (default "auto")
-serial
Run products in sequence rather than concurrently
-since 72h
Collect information within this time. Takes a 'go-formatted' duration, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s)
-terraform-ent
(Experimental) Run Terraform Enterprise diagnostics
-vault
Run Vault diagnostics
-version
Print the current version of hcdiag
Installation
Download the binary with curl.
$ curl \
--silent \
--remote-name \
https://releases.hashicorp.com/hcdiag/0.2.0/hcdiag_0.2.0_linux_amd64.zip
Unzip and remove the archive.
$ unzip hcdiag_0.2.0_linux_amd64.zip && rm -f hcdiag_0.2.0_linux_amd64.zip
Finally add the
hcdiag
executable directory to the PATH
so that you can invoke the command using just its name.$ export PATH=$PWD:$PATH
Here is an example of using hcdiag with Vault
$ hcdiag -vault
2022-02-24T14:50:46.821Z [INFO] hcdiag: Checking product availability
2022-02-24T14:50:46.946Z [INFO] hcdiag: Gathering diagnostics
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=host
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker=stats
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=vault
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker="vault version"
2022-02-24T14:50:46.991Z [INFO] hcdiag: running: seeker="vault status -format=json"
2022-02-24T14:50:47.042Z [INFO] hcdiag: running: seeker="vault read sys/health -format=json"
2022-02-24T14:50:47.088Z [INFO] hcdiag: running: seeker="vault read sys/seal-status -format=json"
2022-02-24T14:50:47.138Z [INFO] hcdiag: running: seeker="vault read sys/host-info -format=json"
2022-02-24T14:50:47.187Z [INFO] hcdiag: running: seeker="vault debug -output=hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz -duration=10s"
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Results.json file: dest=hcdiag-2022-02-24T145046Z/Results.json
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Manifest.json file: dest=hcdiag-2022-02-24T145046Z/Manifest.json
2022-02-24T14:50:58.282Z [INFO] hcdiag: Compressed and archived output file: dest=./hcdiag-2022-02-24T145046Z.tar.gz
List the directory for tar+gzip archive files to discover the file that
hcdiag
created.$ ls -l *.gz
-rw-r--r-- 1 vault vault 139143 Aug 10 21:18 hcdiag-2022-02-15T175841Z.tar.gz
The filename contains a
hcdiag
prefix, followed by a timestamp. You can unpack the archive and further examine its contents.$ tar zxvf hcdiag-2022-02-24T145046Z.tar.gz
hcdiag-2022-02-24T145046Z/Manifest.json
hcdiag-2022-02-24T145046Z/Results.json
hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz
The archive extracts three files into a temporary directory.
-
Manifest.json contains information describing the
hcdiag
run, including configuration options used, run duration, and a count of any errors encountered. -
Results.json contains information about the environment and the output from an invocation of vault debug.
-
VaultDebug.tar.gz contains the output from invoking
vault debug
as described in
You can further explore the
VaultDebug.tar.gz
archive by unpacking it and examining its contents.$ tar zxvf temp005038472/VaultDebug.tar.gz
VaultDebug/
VaultDebug/2021-08-10T21-18-42Z/
VaultDebug/2021-08-10T21-18-42Z/allocs.prof
VaultDebug/2021-08-10T21-18-42Z/block.prof
VaultDebug/2021-08-10T21-18-42Z/goroutine.prof
VaultDebug/2021-08-10T21-18-42Z/goroutines.txt
VaultDebug/2021-08-10T21-18-42Z/heap.prof
VaultDebug/2021-08-10T21-18-42Z/mutex.prof
VaultDebug/2021-08-10T21-18-42Z/profile.prof
VaultDebug/2021-08-10T21-18-42Z/threadcreate.prof
VaultDebug/2021-08-10T21-18-42Z/trace.out
VaultDebug/2021-08-10T21-18-52Z/
VaultDebug/2021-08-10T21-18-52Z/allocs.prof
VaultDebug/2021-08-10T21-18-52Z/block.prof
VaultDebug/2021-08-10T21-18-52Z/goroutine.prof
VaultDebug/2021-08-10T21-18-52Z/goroutines.txt
VaultDebug/2021-08-10T21-18-52Z/heap.prof
VaultDebug/2021-08-10T21-18-52Z/mutex.prof
VaultDebug/2021-08-10T21-18-52Z/threadcreate.prof
VaultDebug/config.json
VaultDebug/host_info.json
VaultDebug/index.json
VaultDebug/metrics.json
VaultDebug/replication_status.json
VaultDebug/server_status.json
VaultDebug/vault.log
Example, to include only 24 hours of log output:
$ hcdiag -vault -since 24h
Troubleshooting
Issue: Error reading logs from the journal
Example:
[ERROR] hcdiag: journalctl -u vault: message="unable to read logs, is your user allowed? try sudo?"
output=
| Hint: You are currently not seeing messages from other users and the system.
| Users in groups 'adm', 'systemd-journal' can see all messages.
| Pass -q to turn off this notice.
| -- No entries --
Solution: Re-run
hcdiag
as a user with permissions to read the journalReferences
For more information about using hcdiag
, please refer to the below documentation: