[Replicated] TFE fails to start with "Ready state command canceled: context deadline exceeded” – HashiCorp Help Center

Problem

After a new installation or upgrade of Terraform Enterprise (TFE) using a mounted disk, the application fails to start. The logs for the terraform-enterprise container show a permission denied error related to a disk check, accompanied by a context deadline exceeded message.

An example of the error log is shown below.

[ERROR] terraform-enterprise: check failed: name=disk duration=\"92.452µs\" err=\"open /opt/terraform-enterprise/data/checkdisk: permission denied\"","component":"terraform-enterprise"}

Cause

This issue occurs when the user running the Terraform Enterprise application does not have the necessary file system permissions to write to the data directory, typically located at /opt/terraform-enterprise/data/. During startup, TFE attempts to create a checkdisk file in this directory to verify write access, and the failure of this operation prevents TFE from starting.

In a correctly configured installation, the directory permissions should appear similar to the following.

drwxr-xr-x 6 root root 4096 Mar 12 12:14 .
drwxr-xr-x 4 root root 4096 Mar 12 12:09 ..
drwxr-xr-x 3 root root 4096 Mar 12 09:31 aux
drwxr-xr-x 2 root root    6 Mar 12 12:07 data
drwxr-xr-x 3 root root 4096 Mar 12 12:13 postgres
drwxr-xr-x 2 root root 4096 Mar 12 12:13 postgres-backup

Solutions

Solution 1: Verify and Correct File System Permissions

To resolve this issue, you must verify and correct the permissions on the TFE data directory.

Connect to the terraform-enterprise container to perform a diagnostic check. Execute the following command on the host machine.
```
$ docker exec -it terraform-enterprise bash
```
Once inside the container, attempt to create a test file in the data directory.
```
$ touch /opt/terraform-enterprise/data/test
```
If you receive a Permission denied error, it confirms an issue with the directory's ownership or permissions.
```
touch: cannot touch 'test': Permission denied
```
Exit the container and, on the host machine, review and adjust the ownership and permissions for the mounted TFE installation path to ensure the user running the TFE services has write access. Consult the official Terraform Enterprise installation documentation for the specific permissions required for your environment.

Outcome

After you correct the file system permissions, Terraform Enterprise should start successfully without the permission denied error.

Additional Information

For detailed requirements, please refer to the official Terraform Enterprise documentation regarding mounted disk installations and file system permissions.