Introduction
Problem
Prior to Consul version 1.9.0, when Consul's HTTP connections rate-limiting threshold is breached, they get silently reset without any kind of error message present in the API calls, nor a message in the Consul logs. This could lead to a potential misdirection of troubleshooting into investigating networking and security solutions on the path of the connections. This can present in the vault logs with messages similar to:
error="Get http://127.0.0.1:8500/v1/.../?keys=&separator=%2F: EOF"
Prerequisites (if applicable)
- A Consul cluster version 1.8.19 or older
- An amount of HTTP API calls to the cluster higher than the configured limit; or higher than the default limit of 200
Solutions:
Raise the connection limit's value (default 200). Below - JSON configuration example:
"limits": {
"http_max_conns_per_client": 300
}
Additional Information
-
Consul documentation for HTTP rate limiting