hcdiag
is a universal troubleshooting data gathering tool you can use to collect and archive important data from HashiCorp product installations. The tool can be used with Vault, Consul and Nomad. The information gathered by hcdiag
is good for incident response and troubleshooting errors.-dryrun
.Prerequisites
Version(s) | Notes |
All | hcdiag needs to be run on the server running the product(s). |
0.3.0 | TLS API limitations are resolved |
0.2.0 and older | Limited TLS API support |
hcdiag Release
Consul
Required Environment Variables
-
CONSUL_HTTP_ADDR set to the address to the local Consul agent
-
CONSUL_TOKEN must be set to the API access token if ACLs are enabled
Tutorial
Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/
Nomad
hcdiag
Required Environment Variables
-
NOMAD_ADDR set to the address of the Nomad server
-
NOMAD_TOKEN must be set if ACLs are enabled
- Must have sufficient permissions. For more information see: https://www.vaultproject.io/docs/commands/debug
Tutorial
Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/
Vault
Required Environment Variables
-
VAULT_ADDR set to the address of the Vault server
-
VAULT_TOKEN must be set to the Vault authentication token
-
Alternatively, a token may also exist at
~/.vault_token
-
If both are present, VAULT_TOKEN will be used
-
Must have sufficient permissions. For more information see: https://www.vaultproject.io/docs/commands/debug
-
Tutorial
Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/
Terraform Enterprise
Required Environment Variables
-
TFE_HTTP_ADDR set to the url of the TFE server
-
export TFE_HTTP_ADDR = https://<hostnameFQDN>
-
-
TFE_TOKEN should be a generated Terraform Enterprise API token
Tutorial
Note: Use tutorial only as a Guiding Example! Ensure the hcdiag binary is the latest version from https://releases.hashicorp.com/hcdiag/
Submitting a ticket to Support
When submitting a ticket to Help Center, please submit hcdiag-{YYYY}-{MM}-{mm}T{nanos}Z.tar.gz. Please only submit the tar ball without renaming it instead of individual log files as Support Engineers would be able to diagnose the issue efficiently.
Usage
Input Flags
Argument
|
Description
|
Type
|
Default Value
|
---|---|---|---|
dryrun
|
Perform a dry run to display commands without executing them
|
bool
|
false
|
os
|
Override operating system detection
|
string
|
"auto"
|
consul
|
Run Consul diagnostics
|
bool
|
false
|
nomad
|
Run Nomad diagnostics
|
bool
|
false
|
vault
|
Run Vault diagnostics
|
bool
|
false
|
includes
|
files or directories to include (comma-separated, file--globbing available if 'wrapped--in-single-quotes') e.g. '/var/log/consul- ,/var/log/nomad- '
|
string
|
""
|
include-since
|
Time range to include files, counting back from now. Takes a 'go-formatted' duration, usage examples: 72h, 25m, 45s, 120h1m90s
|
string
|
"72h"
|
destination
|
Path to the directory the bundle should be written in
|
string
|
"."
|
dest
|
Shorthand for -destination
|
string
|
"."
|
config
|
Path to HCL configuration file
|
string
|
""
|
debug-duration
|
Duration to run product debug bundle commands. Usage examples: 10s, 25m, 2h10m30s | string | "10s" |
debug-interval
|
How long metrics collection intervals in product debug commands last. Usage examples: 10s, 25m, 2h10m30s | string | "5s" |
serial
|
Run products in sequence rather than concurrently. Mostly for dev - use only if you want to be especially delicate with system load.
|
bool
|
false
|
version
|
Print the current version of hcdiag
|
|
|
Examples
-
Host diagnostics only
-
hcdiag
-
Host and Vault diagnostics
-
hcdiag -vault
-
Host, Consul, and Nomad diagnostics
-
hcdiag -consul -nomad
-
Host and all available product diagnostics
-
hcdiag -all
hcdiag --help
as seen below: $ hcdiag --help Usage of hcdiag: -all DEPRECATED: Run all available product diagnostics -config string Path to HCL configuration file -consul Run Consul diagnostics -debug-duration 00h00m00s How long to run product debug bundle commands. Provide a duration ex: 00h00m00s. See: -duration in `vault debug`, `consul debug`, and `nomad operator debug` (default 10s) -debug-interval 00h00m00s How long metrics collection intervals in product debug commands last. Provide a duration ex: 00h00m00s. See: -interval in `vault debug`, `consul debug`, and `nomad operator debug` (default 5s) -dest string Shorthand for -destination (default ".") -destination string Path to the directory the bundle should be written in (default ".") -dryrun Performing a dry run will display all commands without executing them -include-since 72h Alias for -since, will be overridden if -since is also provided, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s) -includes value files or directories to include (comma-separated, file-*-globbing available if 'wrapped-*-in-single-quotes') e.g. '/var/log/consul-*,/var/log/nomad-*' -nomad Run Nomad diagnostics -os string Override operating system detection (default "auto") -serial Run products in sequence rather than concurrently -since 72h Collect information within this time. Takes a 'go-formatted' duration, usage examples: 72h, `25m`, `45s`, `120h1m90s` (default 72h0m0s) -terraform-ent (Experimental) Run Terraform Enterprise diagnostics -vault Run Vault diagnostics -version Print the current version of hcdiag
Installation
$ curl \
--silent \
--remote-name \
https://releases.hashicorp.com/hcdiag/0.3.0/hcdiag_0.3.0_linux_amd64.zip
$ unzip hcdiag_0.3.0_linux_amd64.zip && rm -f hcdiag_0.3.0_linux_amd64.zip
hcdiag
executable directory to the PATH
so that you can invoke the command using just its name.$ export PATH=$PWD:$PATH
$ hcdiag -vault
2022-02-24T14:50:46.821Z [INFO] hcdiag: Checking product availability
2022-02-24T14:50:46.946Z [INFO] hcdiag: Gathering diagnostics
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=host
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker=stats
2022-02-24T14:50:46.946Z [INFO] hcdiag: Running seekers for: product=vault
2022-02-24T14:50:46.946Z [INFO] hcdiag: running: seeker="vault version"
2022-02-24T14:50:46.991Z [INFO] hcdiag: running: seeker="vault status -format=json"
2022-02-24T14:50:47.042Z [INFO] hcdiag: running: seeker="vault read sys/health -format=json"
2022-02-24T14:50:47.088Z [INFO] hcdiag: running: seeker="vault read sys/seal-status -format=json"
2022-02-24T14:50:47.138Z [INFO] hcdiag: running: seeker="vault read sys/host-info -format=json"
2022-02-24T14:50:47.187Z [INFO] hcdiag: running: seeker="vault debug -output=hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz -duration=10s"
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Results.json file: dest=hcdiag-2022-02-24T145046Z/Results.json
2022-02-24T14:50:58.275Z [INFO] hcdiag: Created Manifest.json file: dest=hcdiag-2022-02-24T145046Z/Manifest.json
2022-02-24T14:50:58.282Z [INFO] hcdiag: Compressed and archived output file: dest=./hcdiag-2022-02-24T145046Z.tar.gz
hcdiag
created.$ ls -l *.gz
-rw-r--r-- 1 vault vault 139143 Aug 10 21:18 hcdiag-2022-02-15T175841Z.tar.gz
hcdiag
prefix, followed by a timestamp. You can unpack the archive and further examine its contents.$ tar zxvf hcdiag-2022-02-24T145046Z.tar.gz
hcdiag-2022-02-24T145046Z/Manifest.json
hcdiag-2022-02-24T145046Z/Results.json
hcdiag-2022-02-24T145046Z/VaultDebug.tar.gz
-
Manifest.json contains information describing the
hcdiag
run, including configuration options used, run duration, and a count of any errors encountered. -
Results.json contains information about the environment and the output from an invocation of vault debug.
-
VaultDebug.tar.gz contains the output from invoking
vault debug
as described in
VaultDebug.tar.gz
archive by unpacking it and examining its contents.$ tar zxvf temp005038472/VaultDebug.tar.gz
VaultDebug/
VaultDebug/2021-08-10T21-18-42Z/
VaultDebug/2021-08-10T21-18-42Z/allocs.prof
VaultDebug/2021-08-10T21-18-42Z/block.prof
VaultDebug/2021-08-10T21-18-42Z/goroutine.prof
VaultDebug/2021-08-10T21-18-42Z/goroutines.txt
VaultDebug/2021-08-10T21-18-42Z/heap.prof
VaultDebug/2021-08-10T21-18-42Z/mutex.prof
VaultDebug/2021-08-10T21-18-42Z/profile.prof
VaultDebug/2021-08-10T21-18-42Z/threadcreate.prof
VaultDebug/2021-08-10T21-18-42Z/trace.out
VaultDebug/2021-08-10T21-18-52Z/
VaultDebug/2021-08-10T21-18-52Z/allocs.prof
VaultDebug/2021-08-10T21-18-52Z/block.prof
VaultDebug/2021-08-10T21-18-52Z/goroutine.prof
VaultDebug/2021-08-10T21-18-52Z/goroutines.txt
VaultDebug/2021-08-10T21-18-52Z/heap.prof
VaultDebug/2021-08-10T21-18-52Z/mutex.prof
VaultDebug/2021-08-10T21-18-52Z/threadcreate.prof
VaultDebug/config.json
VaultDebug/host_info.json
VaultDebug/index.json
VaultDebug/metrics.json
VaultDebug/replication_status.json
VaultDebug/server_status.json
VaultDebug/vault.log
$ hcdiag -vault -since 24h
Troubleshooting
Issue: Error reading logs from the journal
Example:
[ERROR] hcdiag: journalctl -u vault: message="unable to read logs, is your user allowed? try sudo?"
output=
| Hint: You are currently not seeing messages from other users and the system.
| Users in groups 'adm', 'systemd-journal' can see all messages.
| Pass -q to turn off this notice.
| -- No entries --
hcdiag
as a user with permissions to read the journalReferences
For more information about using hcdiag
, please refer to the below documentation: