Introduction
Problem
When using Consul-backed storage with Vault (or with integrated raft storage), you can run into a situation where Vault attempts to save data that exceeds the maximum request length configured for Consul. These requests will fail with an error similar to this:
* rpc error: code = Unknown desc = failed to create an entity for the authenticated alias: failed to persist packed storage entry: Failed request: Request body(524401 bytes) too large, max size: 524288 bytes.
Cause
These issues can happen if you have a txn_max_req_len set too low in the Consul configuration (or a max_entry_size configured too low for integrated storage), or if there is some Vault activity that is causing data saved to be unreasonably large. It can also happen in the context of replication if the configured KV size is larger on the primary cluster than the secondary cluster. The error message will be clear about the size of the request vs. the maximum configured size, so based on that information you may need to tune the maximum values and/or investigate what is causing undesirably massive values to be saved in vault.
Solutions:
If the difference between the body size and max configured size is close in value, it's probably enough to simply bump up those configured values, either the txn_max_req_len for Consul, or the max_entry_size for Vault integrated storage.
If the difference is extreme, and the requests contain unreasonably large sets of data, they should be investigated to determine if a workflow change of some kind is in order.
Additional Information:
For more details, please refer to the following relevant documentation pages:
https://www.vaultproject.io/docs/internals/limits
https://www.consul.io/docs/agent/options#txn_max_req_len