Enabling sidecar for a service without an explicit health check causes default TCP health check to be invoked

Problem

Enabling a sidecar for a service without explicit health checks in the sidecar services causes premature connection closures.

Based on the clients, the warning messages may be different.

E.g.,

a) The below warning is recorded in the ZooKeeper logs -

org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /127.0.0.1:41110, session = 0x0

b) The below message appears when iperf3 is used -

iperf3: error - unable to receive parameters from client: Bad file descriptor

Cause:

• Consul Agent spawns a TCP connection every 10s by default to check on the sidecar health.
• The sidecar spawns a connection to the ZooKeeper server because it received an incoming TCP connection.
• The TCP connection is then closed by both the agent and the sidecar.
• Certain clients like ZooKeeper do not handle these TCP port open checks that do not contain the valid payload it expects.
• This connection check gets invoked if there is no explicit health check defined under the sidecar service.

Solution:

• When defining a service with a sidecar_service, consider including an explicit check if needed.

• An example is shown below that includes a check not only at the service level, but also at the sidecar_service level. 

{
  "id": "zookeeper-connect",
  "name": "zookeeper-connect",
  "tags": ["venus", "zookeeper-connect"],
  "address": "10.237.8.148",
  "port": 2181,
  "check": {
    "args": ["/tmp/check_zookeeper.sh", "from_zookeeper-connect-svc-check"],
    "interval": "10s",
    "timeout": "5s"
  },
  "connect": {
    "sidecar_service": {
      "check": {
        "args": ["/tmp/check_zookeeper.sh", "from_zookeeper-connect-sidecar-check"],
        "interval": "10s",
        "timeout": "5s"
      }
    }
  }
}

Outcome

By following the above suggestion, warnings that are generated due to half-open TCP port checks will no longer appear.