How To Troubleshoot Terraform Enterprise Flexible Deployment Options Startup Issues – HashiCorp Help Center

Introduction

This guide provides procedures to debug startup issues for Terraform Enterprise Flexible Deployment Options when the application fails to start and the logs do not provide sufficient detail. The steps involve modifying the container entrypoint to keep it running, which allows for manual execution of the startup script and direct access to log files.

Prerequisites

An existing installation of Terraform Enterprise Flexible Deployment Options using either Docker or Podman.

Procedure

Follow the steps for your specific container environment.

For Docker Environments

Edit the docker-compose.yaml file to add a custom entrypoint. This change prevents the container from exiting immediately, allowing you to perform manual troubleshooting steps.

---
name: terraform-enterprise
services:
  tfe:
    image: images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#>
    ## Adding Custom Entrypoint for Troubleshooting Startup Issues
    entrypoint: "tail -f /dev/null"
    environment:
      TFE_LICENSE: "<Hashicorp license>"
    ## ... truncated

Optionally, modify the docker-compose.yaml file to mount the log directory to the host filesystem, which can simplify log collection for support.
```
## ...
services:
  tfe:
    ## ...
    read_only: false
    tmpfs:
      - /tmp:mode=01777
      - /run
      #- /var/log/terraform-enterprise
```
Deploy the container with the saved changes.
```
$ docker compose up -d
```
Execute the Terraform Enterprise startup script inside the running container.
```
$ docker exec -it terraform-enterprise-tfe-1 /usr/local/bin/supervisord-run
```

After the startup script exits, review the logs to identify the root cause. You can access logs from within the container or directly from the host if you have root access.

## Access logs from within the Docker container
$ docker exec -it -w /var/log/terraform-enterprise terraform-enterprise-tfe-1 sh -c 'ls -l && bash'

## Example output
total 20
-rw-r--r-- 1 terraform-enterprise root 1022 Jul 18 22:27 redis.log
-rw-r--r-- 1 terraform-enterprise root 1197 Jul 18 22:27 supervisord.log
-rw-r--r-- 1 terraform-enterprise root 9941 Jul 18 22:27 terraform-enterprise.log

## Access logs from the Linux host (requires root access)
# cd $(docker inspect terraform-enterprise-tfe-1 --format '{{.GraphDriver.Data.MergedDir }}')/var/log/terraform-enterprise && ls -l

## Example output
total 20
-rw-r--r-- 1 mj root 1022 Jul 18 17:27 redis.log
-rw-r--r-- 1 mj root 1197 Jul 18 17:27 supervisord.log
-rw-r--r-- 1 mj root 9941 Jul 18 17:27 terraform-enterprise.log

Optionally, create a compressed archive of the Terraform Enterprise logs.

## This command requires sudo access
# sudo tar cvzf /path/to/save/terraform-enterprise.logs.tar.gz $(docker inspect terraform-enterprise-tfe-1 --format '{{.GraphDriver.Data.MergedDir }}')/var/log/terraform-enterprise

For Podman Environments

Edit your Kubernetes pod definition file (<kube.yaml>) to add a custom command and arguments. This change keeps the pod running indefinitely so you can perform manual troubleshooting.

---
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    app: "terraform-enterprise"
  name: "terraform-enterprise"
spec:
  restartPolicy: "Never"
  containers:
    - env:
        - name: "TFE_OPERATIONAL_MODE"
      ## ... (truncated)
      image: "images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#>"
      ## Adding Custom Entrypoint for Troubleshooting Startup Issues
      command: [ "/bin/bash", "-c", "--" ]
      args: [ "while true; do sleep 3600; done;" ]
      ## ... (truncated)

Deploy the pod using the modified definition file.
```
$ podman play kube /path/to/<kube.yaml>
```

Execute the Terraform Enterprise startup script inside the running pod.

$ podman exec -it terraform-enterprise-terraform-enterprise /usr/local/bin/supervisord-run

After the startup script exits, review the logs from within the container to identify the root cause.

$ podman exec -it -w /var/log/terraform-enterprise terraform-enterprise-terraform-enterprise sh -c 'ls -l && bash'

## Example output
total 24
-rw-r--r--. 1 terraform-enterprise root 1547 Jul 19 15:28 postgres.log
-rw-r--r--. 1 terraform-enterprise root 1022 Jul 19 15:28 redis.log
-rw-r--r--. 1 terraform-enterprise root 1344 Jul 19 15:28 supervisord.log
-rw-r--r--. 1 terraform-enterprise root 9010 Jul 19 15:28 terraform-enterprise.log

Optionally, create a compressed archive of the logs inside the container and copy it to the host filesystem.

$ podman exec -it terraform-enterprise-terraform-enterprise sh -c 'tar cvzf /tmp/terraform-enterprise.logs.tar.gz /var/log/terraform-enterprise/' && podman cp terraform-enterprise-terraform-enterprise:/tmp/terraform-enterprise.logs.tar.gz /path/to/save/file/

Additional Information

For more details on the checks performed during startup, refer to the official documentation on Terraform Enterprise FDO startup checks.