Introduction
Terraform Enterprise v202404-2 or later fails to start and the following error from the Fluent Bit service is visible in the startup logs.
2024-05-08 14:07:52,962 INFO supervisord started with pid 1
2024-05-08 14:07:53,965 INFO spawned: 'fluent-bit' with pid 28
2024-05-08 14:07:53,967 INFO spawned: 'terraform-enterprise' with pid 29
2024-05-08 14:07:54,243 INFO waiting for fluent-bit, terraform-enterprise to die
2024-05-08 14:07:55,247 INFO success: fluent-bit entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-08 14:07:55,248 INFO stopped: terraform-enterprise (terminated by SIGTERM)
2024-05-08 14:07:55,248 INFO reaped unknown pid 41 (exit status 0)
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
...
[2024/05/08 14:07:56] [error] could not open configuration file, aborting.
2024-05-08 14:07:56,019 INFO stopped: fluent-bit (exit status 1)
Prerequisites
- Terraform Enterprise >= v202404-2
Cause
Workarounds
To work around this issue and investigate the root cause of the startup issue, copy the terraform-enterprise log from the exited container to inspect its content for meaningful errors affecting startup.
Docker
docker cp <TFE_CONTAINER>:/var/log/terraform-enterprise/terraform-enterprise.log .
- Ensure the container is started without a read-only filesystem and
/var/log/terraform-enterpriseis not configured as atmpfsmount, otherwise the docker cp command with fail
Kubernetes/OpenShift/Podman
For Kubernetes, OpenShift, and Podman (if the container has an always or on-failure restart policy), tail the file through an exec command while the container is running and before it exits. Note that this may require a few attempts.
Kubernetes
kubectl exec -n <TFE_NAMESPACE> -ti <TFE_POD> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log
OpenShift
oc exec -n <TFE_NAMESPACE> -ti <TFE_POD> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log
Podman
podman exec <TFE_CONTAINER> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log
Troubleshooting
If the container is exiting too quickly to view the log file, as an alternative approach, modify the configuration so that a stable, interactive shell session can be started in the Terraform Enterprise container. Then, manually invoke the startup script, wait until it exits, and view the generated log files from the shell.
Docker Compose
...
services:
<TFE_SERVICE_NAME>:
stdin_open: true
tty: true
entrypoint: /bin/bash
user: root # (optional)
...
- To attach to running container, run
docker compose attach <TFE_SERVICE_NAME>
Podman
...
spec:
containers:
- command: ["/bin/bash"]
tty: true
stdin: true
...
- To attach to running container, run
podman attach <CONTAINER_ID>
Kubernetes/OpenShift
For Kubernetes, start a debug pod. The following command creates a copy of the Terraform Enterprise pod, overwriting the entrypoint to /bin/bash.
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --copy-to=terraform-enterprise-debug --container=terraform-enterprise -- bash
Start Terraform Enterprise
Once an interactive shell session is started, run the following command to start Terraform Enterprise.
supervisord-run
When the process exits, view the terraform-enterprise log file for startup errors.
cat /var/log/terraform-enterprise/terraform-enterprise.log
terraform-enterpriseprocess at startup as of v202404-2. If the terraform-enterprise process exits early for some reason before creating the file, Fluent Bit fails to start. As a result, it will not perform its function of forwarding logs to the standard output streams of the container and the error indicating the actual cause of the startup failure is not surfaced to the user. To investigate the cause of the terraform-enterprise startup error, refer to the instructions below.