Introduction
By default, when an Azure standard load balancer is created using a Kubernetes service definition it performs health probes for all ports declared in ports section of the service manifest.
If the Azure Standard Load Balancer is also configured to front port 8201, which is used by integrated storage replication and request forwarding, this may result in the following messages being displayed repeatedly in the Vault Operational Logs:
2024-01-29T17:57:17.342Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error=EOF
2024-01-29T17:57:17.342Z [INFO] http: TLS handshake error from XXX.XXX.XXX.XXX:56436: EOF
For communication between Vault servers (port 8201 by default), Vault automatically negotiates an mTLS connection when new servers join the cluster initially via the API address port (8200 by default).
The AKS Standard Load Balancer Health Probe is not able to establish this mTLS connection, which is the reason for the above error.
Expected Outcome
The expected outcome is to prevent the messages from being displayed repeatedly in the Vault Operational Logs.
Prerequisites (if applicable)
- Vault Enterprise Edition
- Azure Kubernetes Service (AKS)
- A public standard load balancer in Azure Kubernetes Service (AKS)
Procedure
Using the following annotation in the service definition file, disables the health probe preventing the issue from occurring:
- service.beta.kubernetes.io/port_{port}_no_probe_rule
The description of this annotation is as follows:
({port} is service port number. When set to true, no health probe rule for this port is generated.)
For example:
"annotations" = {
"service.beta.kubernetes.io/port_8201_no_probe_rule" = "true"
}
Other annotations which could be of relevance for the configuration of the AKS Azure Standard Load Balancer health probes are:
-
service.beta.kubernetes.io/port_{port}_health-probe_num-of-probe -
service.beta.kubernetes.io/port_{port}_health-probe_protocol -
service.beta.kubernetes.io/port_{port}_health-probe_request-path