Introduction

This article addresses frequently asked questions about Consul's KV store, covering topics such as size limits, performance considerations, ACLs, and replication.

Frequently Asked Questions

• Is there a size limit to store secrets on KV v1/v2 secret engine on Vault?

When using Consul as a storage backend, the maximum size for a single key-value entry is 512 KiB by default. This limit is enforced by Consul itself. However, you can configure this limit using the kv_max_value_size parameter, which was introduced in Consul version 1.5.3.

kv_max_value_size (Advanced)

This parameter controls the maximum size (in bytes) allowed for a single key-value entry in Consul's KV store.

• Default: 512 KB (Raft's suggested maximum)
• Impact: Improper tuning (excessively large values) can negatively affect Consul's performance, including leadership stability and heartbeat timings, due to increased I/O overhead.
• Txn Endpoint: This setting also affects the /v1/txn endpoint. However, for finer-grained control over transaction sizes, it's recommended to use the txn_max_req_len parameter introduced in Consul 1.7.2. If both parameters are set, the higher limit takes precedence.

You can see more on viewing and configuring Consul options here

• What is the Suggested Limit on Consul KV store?

Consul's KV store can be a useful tool, but it's important to be mindful of its limitations. Storing excessive amounts of data in the KV store can negatively impact Consul's core functions, such as service discovery and health checking. This can manifest as frequent leadership changes, heartbeat timeouts, and other stability issues. For optimal performance, it's advisable to limit KV usage to small-scale data and explore alternative solutions for larger datasets. You can find more information in the Key/Value (KV) Store documents. 

• After enabling Access Control Lists (ACLs) in Consul, I'm unable to read data from the KV store. Are there specific ACL configurations required for KV access?

ACLs control access to the KV store. Make sure your token has the required permissions. For more information on Consul ACLs and tokens, see the Access Control List (ACL) documents.

• Where does the Consul KV store its data?

Consul uses a consensus protocol called Raft to manage and persist its data. This data, including KV entries, is kept in memory for performance, but Raft ensures that every change is also written to disk (as a snapshot and a change log). This provides durability in case of server restarts. So, while disk storage is important, sufficient RAM is crucial for Consul to operate efficiently.

• How can I replicate my KVs to another datacenter?

You can replicate KV data between Consul datacenters using two methods:

1. KV export and import:
   • Use consul kv export to generate a JSON file containing the KV pairs you want to replicate.
   • Then, use consul kv import in the destination datacenter to import the data from the JSON file.
2. Consul Replicate:
   • • This tool provides a more automated and efficient way to replicate KV data.
     • The consul-replicate daemon runs in the background and continuously synchronizes KV data between datacenters.

Important: When replicating KV data, ensure that the KV entries are within the size limits of the destination datacenter's KV store. Exceeding these limits can cause replication failures or performance issues. If you experience CPU/memory spikes during replication, or if consul-replicate fails to complete, it may indicate that the KV store is overloaded. In such cases, consider increasing the kv_max_value_size parameter in the destination datacenter or reducing the size of the KV entries being replicated.

• Why am I seeing "Request body(XXX bytes) too large" errors in my logs?

The "Request body too large" error usually means you're trying to store data that exceeds Consul's configured limits. Here's how to troubleshoot:

• Consul as storage backend:
  • Check txn_max_req_len (for transactions) and kv_max_value_size (for individual KV entries). Increase these values if needed.
• Vault integration:
  • Investigate if Vault is writing excessively large data to Consul. Review Vault's configuration and audit its activity.
• Replication:
  • Ensure the KV size limits (kv_max_value_size) in the source and destination datacenters are consistent.

For additional help with this sort of error, please refer to the Why am I seeing "Request body(XXX bytes) too large" errors in my logs? article. 

Additional Information

• Key/Value (KV) Store

• Agents Configuration File Reference
• Consensus Protocol - Raft Protocol Overview
• Raft: Understandable Distributed Consensus
• Why am I seeing "Request body(XXX bytes) too large" errors in my logs?