Terraform Enterprise >= v202404-2 Fails to Start With Error "could not open configuration file, aborting."

Problem

When starting Terraform Enterprise version v202404-2 or later, the application fails to start. The startup logs show an error from the Fluent Bit service, and the terraform-enterprise process terminates.

2024-05-08 14:07:53,965 INFO spawned: 'fluent-bit' with pid 28
2024-05-08 14:07:53,967 INFO spawned: 'terraform-enterprise' with pid 29
## ...
2024-05-08 14:07:55,248 INFO stopped: terraform-enterprise (terminated by SIGTERM)
## ...
[2024/05/08 14:07:56] [error] could not open configuration file, aborting.
2024-05-08 14:07:56,019 INFO stopped: fluent-bit (exit status 1)

Prerequisites

• Terraform Enterprise version v202404-2 - v202411-1

Cause

The error could not open configuration file indicates a startup failure in the terraform-enterprise process, not in Fluent Bit itself.

As of version v202404-2, the terraform-enterprise process templates and creates the Fluent Bit configuration file at startup. If the terraform-enterprise process exits prematurely due to an unrelated error, it never creates this file, causing Fluent Bit to fail. Because Fluent Bit is responsible for forwarding logs to the container's standard output, the actual error causing the terraform-enterprise process to fail is not visible in the main container logs.

To find the root cause, you must inspect the terraform-enterprise.log file directly inside the container.

Solutions

There are several methods to access the terraform-enterprise.log file to diagnose the underlying startup error.

Solution 1: Copy Logs from an Exited Container (Docker)

If you are using Docker, you can copy the log file from the exited container to your local machine to inspect its contents.

Note: This method requires that the container's filesystem is not read-only and that /var/log/terraform-enterprise is not configured as a tmpfs mount.

1. Copy the log file from the container.

   $ docker cp <TFE_CONTAINER>:/var/log/terraform-enterprise/terraform-enterprise.log .

2. Open the local terraform-enterprise.log file to find the startup error.

Solution 2: Tail Logs from a Running Container (Kubernetes, OpenShift, Podman)

For environments like Kubernetes, OpenShift, or Podman where containers have a restart policy (always or on-failure), you can tail the log file while the container is briefly running before it exits. This may require a few attempts to execute the command at the right time.

• Kubernetes

  $ kubectl exec -n <TFE_NAMESPACE> -ti <TFE_POD> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log

• OpenShift

  $ oc exec -n <TFE_NAMESPACE> -ti <TFE_POD> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log

• Podman

  $ podman exec <TFE_CONTAINER> -- tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log

Solution 3: Inspect Logs via an Interactive Shell

If the container exits too quickly to inspect the log file, you can modify its configuration to start an interactive shell session instead of the default entrypoint. This allows you to manually start the application and inspect the logs from within the container.

1. Modify your deployment configuration to start an interactive shell.

   • Docker Compose

     Update your docker-compose.yml file.

     ## ...
     services:
       <TFE_SERVICE_NAME>:
         stdin_open: true
         tty: true
         entrypoint: /bin/bash
         user: root # (optional)
     ## ...

     Attach to the running container.

     $ docker compose attach <TFE_SERVICE_NAME>

   • Podman

     Update your container specification.

     ## ...
     spec:
       containers:
         - command: ["/bin/bash"]
           tty: true
           stdin: true
     ## ...

     Attach to the running container.

     $ podman attach <CONTAINER_ID>

   • Kubernetes/OpenShift

     Start a debug pod, which creates a copy of your Terraform Enterprise pod with the entrypoint overridden to /bin/bash.

     $ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --copy-to=terraform-enterprise-debug --container=terraform-enterprise -- bash

2. Once you have an interactive shell session, manually start Terraform Enterprise.

   $ supervisord-run

3. After the process exits, view the log file to identify the startup error.

   $ cat /var/log/terraform-enterprise/terraform-enterprise.log

Additional Information

• For more details on log inspection, refer to the documentation on Inspecting logs in flexible deployments.