Introduction
Problem
When the Terraform Enterprise container starts on Kubernetes and there is a configuration error the pod crashes before you have time to look at the logs in detail.
Prerequisites
- Terraform Enterprise FDO on Kubernetes
Cause
The containers starts and by default most logs should be visible with the default Kubernetes logs command
kubectl -n <NAMESPACE> logs terraform-enterprise
With certain configuration issues this not working correctly and doesn't show all the logs you would like. To see these you would need to login to the pod and view the specific log. For example do the following steps
# Go into the container
kubectl -n <NAMESPACE> exec -it terraform-enterprise -- bash
# Go to the logs directory
$ cd /var/log/terraform-enterprise
# view the logs and investigate a specific log
$ ls
archivist.log
atlas.log
backup-restore.log
licensing.log
metrics.log
nginx-access.log
nginx-error.log
nginx.log
outbound-http-proxy.log
sidekiq.log
slug-ingress.log
supervisord.log
task-worker.log
terraform-enterprise.log
terraform-registry-api.log
terraform-registry-worker.log
terraform-state-parser.log
tfe-health-check.log
vault.log
# view a specific logfile
$ cat terraform-enterprise.log
If the pod keeps crashing you only have a few seconds to do these steps.
Solutions:
Use the following script that you can start in a another terminal. This will check every 2 seconds if a pod in the namespace is started and tries to connect and show the specific logfile
#!/bin/bash
# Specify the namespace you want to monitor
NAMESPACE="terraform-enterprise"
# Specify the pods to ignore
IGNORED_PODS=("another-pod" "another-pod2" "another-pod3")
# Specify the logfile to tail. Default is terraform-enterprise.log which should contain the logs you need
LOGFILE="terraform-enterprise.log"
# Function to get the name of the first pod found in the namespace, excluding the ignored pods
get_pod_name() {
PODS=$(kubectl get pods -n "$NAMESPACE" --no-headers -o custom-columns=":metadata.name")
for pod in "${IGNORED_PODS[@]}"; do
PODS=$(echo "$PODS" | grep -v "$pod")
done
echo "$PODS" | head -n 1
}
# Loop until a pod is detected
while true; do
POD_NAME=$(get_pod_name)
if [ -n "$POD_NAME" ]; then
echo "Pod '$POD_NAME' detected in namespace '$NAMESPACE'. Fetching logs:"
kubectl -n "$NAMESPACE" exec -it "$POD_NAME" -- tail -f /var/log/terraform-enterprise/"$LOGFILE"
break
else
echo "No pods found in namespace '$NAMESPACE' (excluding ignored pods). Retrying in 2 seconds..."
sleep 2
fi
done
Outcome
The moment the pod start it should show the logs immediately for the logfile you specified
Pod 'terraform-enterprise-6db8847d65-g4w65' detected in namespace 'terraform-enterprise'. Fetching logs:
2024-10-10T14:07:57.839Z [INFO] terraform-enterprise: connected successfully to terraform_enterprise database