Introduction
This article addresses an issue in Consul versions prior to 1.9.0, where exceeding the HTTP connection rate limit results in silent connection resets without any error messages, potentially leading to misdirected troubleshooting efforts.
Problem
When the HTTP connections rate limit in Consul is exceeded in versions before 1.9.0, the connections are silently reset without any error message in API calls or the Consul logs. This behavior can cause confusion, leading users to investigate network and security layers instead of identifying the root cause in Consul.
Prerequisites
-
Consul cluster running version 1.8.19 or older
-
A high volume of HTTP API calls exceeding the configured limit or the default limit of 200
Cause
When the number of HTTP API requests surpasses the threshold, Consul versions before 1.9.0 do not return an explicit error message. Instead, connections are simply reset, making it appear as if there is a network or firewall issue.
Consul 1.8.19 and older
When the HTTP connection limit is exceeded, a rate-limited API call results in an empty reply from the server:
Successful call:
[krastin:~] % curl --header "X-Consul-Token: secret" -vvv "localhost:8503/v1/status/leader"
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8503 (#0)
> GET /v1/status/leader HTTP/1.1
> Host: localhost:8503
> User-Agent: curl/7.64.1
> Accept: */*
> X-Consul-Token: secret
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
< Date: Tue, 25 Jan 2022 10:47:54 GMT
< Content-Length: 17
<
* Connection #0 to host localhost left intact
"172.23.0.3:8300"
* Closing connection 0
Rate-limited API call:
[krastin:~] % curl --header "X-Consul-Token: secret" -vvv "localhost:8503/v1/status/leader"
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8503 (#0)
> GET /v1/status/leader HTTP/1.1
> Host: localhost:8503
> User-Agent: curl/7.64.1
> Accept: */*
> X-Consul-Token: secret
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection 0
Consul 1.9.0 and later
Explicitly return an HTTP 429 (Too Many Requests) response with a clear message:
Successful API call:
[krastin:~] % curl --header "X-Consul-Token: secret" -vvv "localhost:8503/v1/status/leader"
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8503 (#0)
> GET /v1/status/leader HTTP/1.1
> Host: localhost:8503
> User-Agent: curl/7.64.1
> Accept: */*
> X-Consul-Token: secret
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
< X-Consul-Default-Acl-Policy: deny
< Date: Tue, 25 Jan 2022 10:40:42 GMT
< Content-Length: 17
<
* Connection #0 to host localhost left intact
"172.23.0.3:8300"
* Closing connection 0
Rate-limited API call:
[krastin:~] % curl --header "X-Consul-Token: secret" -vvv "localhost:8503/v1/status/leader"
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8503 (#0)
> GET /v1/status/leader HTTP/1.1
> Host: localhost:8503
> User-Agent: curl/7.64.1
> Accept: */*
> X-Consul-Token: secret
>
< HTTP/1.1 429 Too Many Requests
< Content-Type: text/plain
< Content-Length: 81
< Connection: close
<
Your IP is issuing too many concurrent connections, please rate limit your calls
* Closing connection 0
Overview of Possible Solution
To mitigate this issue, users can adjust the HTTP connection limits in Consul's configuration.
Solutions
Increase the HTTP connection limit - Adjust the http_max_conns_per_client parameter in the Consul configuration file to increase the limit. Below is an example JSON configuration:
"limits": {
"http_max_conns_per_client": 300
}This increases the connection limit from the default 200 to 300, reducing the likelihood of silent resets.
Outcome
After implementing this configuration change, the Consul will allow more concurrent HTTP connections per client before enforcing rate limits. If users still experience issues, they should confirm the change has been applied correctly and monitor API call patterns to ensure they remain within the configured limits.
Additional Information
-
Consul v.1.9.0 changelog
- Consul documentation for HTTP rate limiting