These are errors which can be encountered when operating Vault Enterprise and Vault Enterprise + HSM servers. This is not an exhaustive list, and will be updated periodically.
HSM Related Errors
Incorrect Slot PIN
If the value for slot specified in the pkcs11 seal stanza configuration is incorrect, you’ll encounter an error like this one:
Error parsing HSM configuration: error when testing HSM configuration: error logging in to HSM: pkcs11: 0xA0: CKR_PIN_INCORRECT
Missing libtool ltdl Library
Vault Enterprise+HSM requires that the OS have the GNU Libtool Dynamic Module Loader library (ltdl) installed. If it is not, you’ll encounter an error like this when you attempt to start Vault:
vault: error while loading shared libraries: libltdl.so.7: cannot open shared object file: No such file or directory
This can be resolved by simply installing the correct package for the OS distribution in question.
- For Debian based distributions, use the libtdl or libtdl7 package
- For RHEL based distributions, use the libtool-ltdl package
High Availability Errors
Error creating token: Error making API request.
URL: POST http://localhost:8200/v1/auth/token/create Code: 500. Errors:
* node not active but active node not found
This error is indicative of a bad HA configuration. Be sure to examine all details of the HA specific configuration bits such as cluster_addr or redirect_addr values and make certain that they’re correct.
Storage Backend Errors
PostgreSQL Storage Backend Errors
If you're attempting to bootstrap a Vault instance against a PostgreSQL storage backend and the PostgreSQL version is <= 9.5, you could encounter the following error (for example upon issuing vault init):
$ vault operator init
* barrier configuration saving failed: failed to write seal configuration: pq: function vault_kv_put(unknown, unknown, unknown, unknown) does not exist
This means that the required function has not yet been created in PostgreSQL; an example of this function appears in the PostgreSQL Storage Backend documentation.
Unable to Connect to Consul
This error could be seen during startup of Vault:
Error detecting redirect address: Get http://localhost:8500/v1/agent/self: dial tcp 10.196.128.69:8500: getsockopt: no route to host Error initializing core: missing redirect address
The above error often indicates a network issue is at play; ensure that there are no firewalls, iptables, etc. rules which could be preventing Vault from reaching Consul.
HTTPS Response from HTTP Request
If a Vault server is configured to use TLS, but the user accessing it is not configuring for HTTPS (i.e. has VAULT_ADDR set to http://... then an error like this will be the result of attempting to communicate with Vault using mismatched protocol:
Error validating token: Get http://vault1.consul:8200/v1/auth/token/lookup-self: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
This was during a token self lookup / authentication operation, so the exact error message could be slightly different, but the key part demonstrating HTTPS response to HTTP request is in this bit: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
TLS Certificate Errors
When running the vault command against a TLS enabled Vault instance, the following error could be encountered: x509: certificate signed by unknown authority
When the Vault server's certificate is self-signed by an internal CA, you will need that CA's certificate passed to the vault command via the -ca-path or -ca-cert options or with the environment variables VAULT_CACERT and VAULT_CAPATH environment variables.
Another option is to add the Vault certificate to the trust store, such as through this process (which is a RHEL specific example):
sudo update-ca-trust enablesudo cp /etc/vault/ssl/certs/vault.crt /etc/pki/ca-trust/source/anchors/ $ sudo update-ca-trust extract
Consul Leadership Issues
If the Vault instance is using Consul for the backend and you observe errors like:
Aug 9 06:43:14 ip-205-203-163 vault: 2017/08/09 06:43:14.568102 [ERROR] core: clearing leader advertisement failed: error=Unexpected response code: 500 (rpc error: rpc error: leadership lost while committing log)
This is indicative of an overwhelmed Consul cluster or one that is experiencing leader election issues.
RPC Request Forwarding Issues
When Vault cannot forward RPC requests, you’ll observe errors like these in the operational logs:
Aug 8 18:00:52 ip-205-203-142 vault: 2017/08/08 18:00:52.296713 [ERROR] core: error during forwarded RPC request: error=r pc error: code = Unavailable desc = grpc: the connection is unavailable
Aug 8 18:00:52 ip-205-203-142 vault: 2017/08/08 18:00:52.296735 [ERROR] http/handleRequestForwarding: error forwarding re quest: error=error during forwarding RPC request
Aug 8 18:00:52 ip-205-203-142 vault: 2017/08/08 18:00:52.296746 [ERROR] core: error during forwarded RPC request: error=r pc error: code = Unavailable desc = grpc: the connection is unavailable
Aug 8 18:00:52 ip-205-203-142 vault: 2017/08/08 18:00:52.296762 [ERROR] http/handleRequestForwarding: error forwarding re quest: error=error during forwarding RPC request
Initialization Failures
vault: 2017/08/09 07:13:58.101603 [ERROR] core: barrier init check failed: error=failed to check for initialization: Unexpected response code: 500
This could indicate a problem with the Vault storage and should be investigated as such.
Leadership Loss
When Consul becomes unstable or unavailable, a class of leadership issue errors like this will often repeat through the Vault logs:
Aug 9 06:31:10 ip-205-203-163 vault: 2017/08/09 06:31:10.266246 [ERROR] core: failed to acquire lock: error=failed to read lock: Unexpected response code: 500
Aug 9 06:41:24 ip-205-203-163 vault: 2017/08/09 06:41:24.289799 [ERROR] index: failed to persist checkpoint: error=failed to persist: Unexpected response code: 500 (rpc error: No cluster leader)
Aug 9 06:41:24 ip-205-203-163 vault: 2017/08/09 06:41:24.289832 [ERROR] merkle.periodic: failed to save checkpoint: error=failed to persist checkpoint: failed to persist: Unexpected response code: 500 (rpc error: No cluster leader)
Aug 9 06:41:32 ip-205-203-163 vault: 2017/08/09 06:41:32.702100 [ERROR] index: failed to persist checkpoint: error=failed to persist: Unexpected response code: 500 (rpc error: No cluster leader)
Aug 9 06:41:32 ip-205-203-163 vault: 2017/08/09 06:41:32.702134 [ERROR] merkle.periodic: failed to save checkpoint: error=failed to persist checkpoint: failed to persist: Unexpected response code: 500 (rpc error: No cluster leader)
We see here that Vault is attempting a new session with Consul, but fails to acquire a lock, fails to persist checkpoint data and so on. Note the numerous response code: 500 and No cluster leader entries above.
Secrets Engine Errors
Missing Secrets Engine
If you attempt a kv operation against a non-existent kv secrets engine, you will encounter an error like the one shown in this example:
$ vault kv put secret/team-one/test1 foo=bar
Error making API request.
URL: GET https://127.0.0.1:8200/v1/sys/internal/ui/mounts/secret/team-one/test1
Code: 403. Errors:
* preflight capability check returned 403, please ensure client's policies grant access to path "secret/team-one/test1/"
You can confirm this is the issue with the output of vault secrets list:
$ vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_58c744b9 per-token private secret storage
identity/ identity identity_d48f5227 identity store
sys/ system system_6739abb3 system endpoints used for control, policy and debugging
vaultron-aws/ aws aws_de2c0413 Vaultron example AWS secrets engine
vaultron-consul/ consul consul_e3900e9d Vaultron example Consul secrets engine
vaultron-kv/ kv kv_bac2c8fc Vaultron example KV version 1 secrets engine
vaultron-root-int/ pki pki_84e6ac74 Vaultron example PKI secrets engine (for int CA)
vaultron-root-pki/ pki pki_dbbceb77 Vaultron example PKI secrets engine (for root CA)
vaultron-ssh-client-signer/ ssh ssh_d5cc13bb Vaultron example SSH Secrets Engine (client)
vaultron-ssh-host-signer/ ssh ssh_90429147 Vaultron example SSH Secrets Engine (host)
vaultron-totp/ totp totp_b3d16e46 Vaultron example TOTP Secrets Engine
vaultron-transit/ transit transit_a408d234 Vaultron example Transit secrets engine
There is not currently a secrets engine enabled at the path secret/ which is the causes of the error.