Introduction
When using Consul as a storage backend for Vault, or Integrated Storage (Raft) it is possible the data attempting to be saved to storage exceeds the maximum request length configured. These requests will fail with an error similar to:
* rpc error: code = Unknown desc = failed to create an entity for the authenticated alias: failed to persist packed storage entry: Failed request: Request body(524401 bytes) too large, max size: 524288 bytes.
Cause
- Consul is used as a storage backend and have a
txn_max_req_lenorkv_max_value_sizeset lower than required. - You use Vault's Integrated Storage backend and have a
max_entry_sizeset lower than required. - Due to some Vault activity causing data to be saved that is unreasonably large.
- In the context of replication, if the KV size configurations are larger on the primary cluster than the secondary cluster.
The error message will be clear about the size of the request versus the maximum configured size. Based on that information, it may be required to tune the maximum values and/or investigate what is causing the excessively large values to be saved in Vault.
Solution
If the difference between the body size and max configured size is close, it is most likely enough to simply increase the configured values:
- txn_max_req_len for Consul 1.7.2 or later
- kv_max_value_size for Consul 1.7.1 or earlier
- max_entry_size for Vault Integrated Storage
If the difference is extreme, and the requests contain unreasonably large sets of data, they should be investigated to determine if a workflow change of some kind is in order.
Keeping in mind that the larger this value is, increases the risk of introducing IO delays across the cluster. Which can potentially lead to leadership instability across the Consul nodes, and in turn can have an impact on Vault stability.