Beginning with Consul v1.15.x, it is possible to configure Consul servers with global RPC rate limits to mitigate the potential issues arising from an excessive number of RPC calls made by clients to Consul resources. These limits can be independently defined for
writerequests. In this context, a read request pertains to any request that does not alter Consul's internal state, whereas a write request encompasses any request that does modify this internal state.
- Enable the Server RPC rate limiting feature.
- Understand the method to establish a baseline for Server RPC calls.
- Differentiate between various rate limit modes.
- Running Consul version v1.15.x or above.
- Determining the baseline number for the amount of RPC calls a server agent receives during normal operation.
- Setting up the Server RPC rate limits to shield from unexpected spikes in RPC calls from the agents.
- See Set a global limit on traffic rates for additional information
- As an aid in troubleshooting certain issues.
As an example: If a necessity arises to record the source IP addresses of the client agents making a certain RPC call, perform the following actions
Note: Setting limits per source IP requires Consul Enterprise.
writelimits to 0 in the following fashion
mode = "permissive"
read_rate = 0
write_rate = 0
consul reloadto have the server apply the modified limits.request_limits
Monitor Consul logs at a debug level using:
consul monitor -log-level debug
Repeat the actions pertinent to the troubleshooting of the issue.
Located the source IP address (
source_addr) from the log lines containing the following string
[DEBUG] agent.server.rpc-rate-limit:Log example:
2023-06-16T09:37:34.160-0700 [DEBUG] agent.server.rpc-rate-limit: RPC exceeded allowed rate limit: rpc=KVS.Apply source_addr=192.168.64.136:57521 limit_type=global/write limit_enforced=false
- Set the
- As an example: If a necessity arises to record the source IP addresses of the client agents making a certain RPC call, perform the following actions
Enforcing the rate limits
- Add the below reloadable HCL section to the Consul server configuration
write_rateshould be determined by the observations completed through monitoring, therefore the below example may not be correct for the specific environment being configured
mode = "enforcing"
read_rate = 100000
write_rate = 100000
- Run the
consul reloadcommand on the server to make use of the added/modified parameter values.
The values for limits.request_limits.mode key could be one of the following:
- permissive: The rate limiter allows requests even if the limits are reached, while still generating metrics and logs to aid operators in understanding the Consul load and fine-tuning the limits.
- enforcing: In this mode, the rate limiter denies requests that exceed the configured rate. Consul generates metrics and logs to assist operators in understanding the load on their Consul environment and adjusting the limits accordingly
- disabled: This is the default mode. This mode disables the rate limiter entirely. All requests are allowed without generating logs or metrics.