Problem
When running Terraform Enterprise with Flexible Deployment Options (FDO) in Kubernetes, the application pod may report failed readiness probes during startup. The output of kubectl describe pod shows warnings similar to the following.
$ kubectl describe pod terraform-enterprise-77f744d7b8-4ggnr -n terraform-enterprise Name: terraform-enterprise-77f744d7b8-4ggnr Namespace: terraform-enterprise Priority: 0 Service Account: terraform-enterprise ## ... (output trimmed for brevity) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 54m default-scheduler Successfully assigned terraform-enterprise/terraform-enterprise-77f744d7b8-4ggnr to aks-default-14506778-vmss000000 Normal Pulling 54m kubelet Pulling image "images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202312-1" Normal Pulled 54m kubelet Successfully pulled image "images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202312-1" in 374.473971ms (374.484771ms including waiting) Normal Created 54m kubelet Created container terraform-enterprise Normal Started 54m kubelet Started container terraform-enterprise Warning Unhealthy 54m (x3 over 54m) kubelet Readiness probe failed: Get "http://10.244.1.4:8080/_health_check": dial tcp 10.244.1.4:8080: connect: connection refused Warning Unhealthy 51m (x18 over 54m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 502
Cause
This behavior is expected because the Terraform Enterprise application startup process is not instantaneous. Kubernetes uses two main types of probes:
- Liveness probes: Determine if an application container is running.
- Readiness probes: Determine if an application is ready to accept traffic.
During startup, the Terraform Enterprise container is live but not yet ready to serve traffic. As a result, the readiness probe fails, and Kubernetes correctly prevents routing traffic to the pod until it passes the health check.
Solution
No action is required, as the failed readiness probes are a normal part of the startup sequence. These warnings should resolve automatically within a few minutes once the application is fully initialized and ready to receive traffic.
If the readiness probe continues to fail for an extended period, it may indicate an underlying application error that requires further investigation.
Additional Information
- For more details on Kubernetes probes, refer to the official documentation on liveness, readiness, and startup probes.
- For more information on FDO, refer to the Terraform Enterprise Flexible Deployment Options documentation.