Introduction
Prior to Consul version 1.9.0, when Consul's HTTP connections rate-limiting threshold is breached, they will silently reset without any error message present in the API calls, nor a message in the Consul logs. This could lead to a potential misdirection of troubleshooting into investigating network and security solutions on the path of the connections.
Problem
Vault logs can present the following error message:
error="Get http://127.0.0.1:8500/v1/.../?keys=&separator=%2F: EOF"
Prerequisites (if applicable)
- A Consul version older than 1.8.19
- Amount of HTTP API calls to the cluster is higher than the configured limit (200 by default)
Solution
Raise the connection limit's value. Below is a JSON configuration example:
"limits": {
"http_max_conns_per_client": 300
}
Additional Information
-
Consul Rate Limiting: https://developer.hashicorp.com/consul/docs/agent/config/config-files#http_max_conns_per_client