Introduction

Expected Outcomes

1. Enable the Server RPC rate limiting feature.
2. Understand the method to establish a baseline for Server RPC calls.
3. Differentiate between various rate limit modes.

Prerequisites

• Running Consul version v1.15.x or above.

Use Case

1. Determining the baseline number for the amount of RPC calls a server agent receives during normal operation.
   • See Initialize rate limit settings to see an example of setting the mode to permissive
   • See Monitor traffic rate limit data for additional information on observing rate limit data
2. Setting up the Server RPC rate limits to shield from unexpected spikes in RPC calls from the agents.
   • See Set a global limit on traffic rates for additional information 
3. As an aid in troubleshooting certain issues.
   • As an example: If a necessity arises to record the source IP addresses of the client agents making a certain RPC call, perform the following actions
     Note: Setting limits per source IP requires Consul Enterprise.
     
     1. Set the read & write limits to 0 in the following fashion
        

         limits {
           request_limits {
             mode = "permissive"
             read_rate = 0
             write_rate = 0
           }
         }

     2. Perform a consul reload to have the server apply the modified limits.request_limits
     3. Monitor Consul logs at a debug level using: consul monitor -log-level debug
     4. Repeat the actions pertinent to the troubleshooting of the issue.
     5. Located the source IP address (source_addr) from the log lines containing the following string
        

        [DEBUG] agent.server.rpc-rate-limit:

        Log example:

         2023-06-16T09:37:34.160-0700 [DEBUG] agent.server.rpc-rate-limit: RPC exceeded allowed rate limit: rpc=KVS.Apply source_addr=192.168.64.136:57521 limit_type=global/write limit_enforced=false

Procedure

Enforcing the rate limits

1. Add the below reloadable HCL section to the Consul server configuration
   Note: The read_rate & write_rate should be determined by the observations completed through monitoring, therefore the below example may not be correct for the specific environment being configured
   

    limits {
      request_limits {
        mode = "enforcing"
        read_rate = 100000
        write_rate = 100000
      }
    }

2. Run the consul reload command on the server to make use of the added/modified parameter values.

• permissive: The rate limiter allows requests even if the limits are reached, while still generating metrics and logs to aid operators in understanding the Consul load and fine-tuning the limits.
• enforcing: In this mode, the rate limiter denies requests that exceed the configured rate. Consul generates metrics and logs to assist operators in understanding the load on their Consul environment and adjusting the limits accordingly
• disabled: This is the default mode. This mode disables the rate limiter entirely. All requests are allowed without generating logs or metrics. 

Additional Information

• Traffic rate limiting overview
• In the event an HTTP request gets denied due to rate limiting, review the section Request denials on the Limit traffic rates overview page (this page is only available when selecting v1.15.x).
• To review the metrics set up for monitoring Consul Server RPC rate limiting, please refer to the Set a global limit on traffic rates page