Introduction
Terraform Enterprise v202404-2 or later fails to start and the following error from the Fluent Bit service is visible in the startup logs.
2024-05-08 14:07:52,962 INFO supervisord started with pid 1
2024-05-08 14:07:53,965 INFO spawned: 'fluent-bit' with pid 28
2024-05-08 14:07:53,967 INFO spawned: 'terraform-enterprise' with pid 29
2024-05-08 14:07:54,243 INFO waiting for fluent-bit, terraform-enterprise to die
2024-05-08 14:07:55,247 INFO success: fluent-bit entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-05-08 14:07:55,248 INFO stopped: terraform-enterprise (terminated by SIGTERM)
2024-05-08 14:07:55,248 INFO reaped unknown pid 41 (exit status 0)
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
...
[2024/05/08 14:07:56] [error] could not open configuration file, aborting.
2024-05-08 14:07:56,019 INFO stopped: fluent-bit (exit status 1)
Prerequisites
- Terraform Enterprise >= v202404-2
Cause
Workarounds
To work around this issue, you can copy the terraform-enterprise log from the exited container to inspect its content for meaningful errors affecting startup.
Docker
docker cp <TFE_CONTAINER>:/var/log/terraform-enterprise/terraform-enterprise.log .
- Ensure the container is started without a read-only filesystem and /var/log/terraform-enterprise is not configured as a tmpfs mount, otherwise the docker cp command with fail
Kubernetes/Podman
For Kubernetes, Podman and Docker (if the container has an always or on-failure restart policy), tail the file through an exec command while the container is running.
kubectl exec -n <TFE_NAMESPACE> -ti <TFE_POD> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log
/run/terraform-enterprise/fluent-bit/fluent-bit.conf, however this file does not initially exist- it is templated and created by theterraform-enterpriseprocess which is started concurrently. In v202404-2 and later, the work of setting up template files was delegated to the terraform-enterprise process. As a result, if exits early and is unable to complete this service setup, its logs are not written to container logs and instead are only available from its log file in the container since Fluent Bit is unable to complete startup. This particular behavior has been identified as a bug and will be addressed by HashiCorp in order to provide a smoother troubleshooting experience.