Un-mounting Secrets Engine With Many Secrets Times Out

Introduction

You may find yourself needing to disable a Vault secrets engine that has a large amount of data associated with it.  Depending on the amount of data, and the resources available to the Vault service, this request may timeout.  Such a timeout would look like the following.

$ vault secrets disable /pki
Error disabling secrets engine at pki/: context deadline exceeded

If you were to "time" this operation, you would find that it times out right after 60 seconds.

$ time (vault secrets disable /pki)
Error disabling secrets engine at pki/: context deadline exceeded

real    1m0.053s
user    0m0.041s
sys 0m0.012s

With Vault, you're able to tweak values and allow for long-running commands to succeed and avoid the timeout context deadline exceeded error.

Procedure

The reason this operation times out after 60 seconds is that we are hitting the VAULT_CLIENT_TIMEOUT limit, which by default is set to 60 seconds.  To allow our operation to run longer, we need to increase this value. To do this, we export VAULT_CLIENT_TOKEN setting it to a higher value.  The steps to do this are demonstrated below:

$ export VAULT_CLIENT_TIMEOUT=300s
$ time (vault secrets disable /pki)
Error disabling secrets engine at pki/: Error making API request.

URL: DELETE http://0.0.0.0:8200/v1/sys/mounts/pki
Code: 400. Errors:

* context canceled

real    1m30.079s
user    0m0.051s
sys 0m0.007s

Here we can see that the error has actually changed structure. Additionally, the command only ran for 90 seconds, not the configured 300 seconds as expected. Why would this be?  The answer is that we are actually hitting another separate configurable timeout, max_request_duration. This value is configurable in the TCP listener stanza, and it specifies the maximum request duration allowed before Vault cancels the request. We can change this value by updating Vault's configuration file and reloading Vault, then repeat our test.

Below is an example of a configuration file with the change made:

$ cat /etc/vault.d/vault.hcl
...
listener "tcp" {
  address         = "0.0.0.0:8200"
  tls_disable     = true
  max_request_duration = "300s"
}
...

And below is a repeat of the test:

$ export VAULT_CLIENT_TIMEOUT=300s
$ time (vault secrets disable /pki)
Success! Disabled the secrets engine (if it existed) at: pki/

real    3m8.060s
user    0m0.064s
sys 0m0.000s

Here we can see that the command has successfully completed, in just over 3 minutes.  By updating these values, we were able to let the process run long enough to complete.