Terraform Enterprise not coming up after update.

Problem

After upgrading Terraform Enterprise (TFE), the application fails to start, and its Docker containers enter a continuous restart loop. When inspecting the replicated service logs, you find a 'permission denied' error related to a PostgreSQL directory.

To view the logs, run the following command.

# journalctl -u replicated

The output contains the following error message.

Failed to migrate retraced events: wait for retraced start: ensure retraced started: start postgres: ensure postgres ownership: mkdir /var/lib/replicated/retraced/postgres/run: mkdir /var/lib/replicated/retraced/postgres/run: permission denied

Prerequisites

• A self-hosted Terraform Enterprise installation.
• Root or sudo access to the TFE instance.

Cause

The error occurs because the file and directory permissions for /var/lib/replicated/retraced/postgres are incorrect. The user and group ownership must be set to replicated:docker for the TFE services to have the necessary access to start the internal PostgreSQL database.

Solution

To resolve this issue, you must verify and correct the directory ownership.

1. Verify the current permissions of the postgres directory. The owner and group should be replicated and docker, respectively.

   $ ls -la /var/lib/replicated/retraced/

   The expected output should show replicated docker for the postgres directory.

   total 16
   drwxr-xr-x  4 replicated docker 4096 Jan 12 16:22 .
   drwxr-xr-x 15 replicated docker 4096 Jan 12 16:23 ..
   drwxr-xr-x  2 replicated docker 4096 Jan 12 16:22 api
   drwxr-xr-x  4 replicated docker 4096 Jan 12 16:22 postgres

2. If the ownership is incorrect, change the owner and group recursively for the postgres directory.

   $ chown -R replicated:docker /var/lib/replicated/retraced/postgres

After correcting the permissions, the Terraform Enterprise services should start successfully.

Additional Information

For more details on administering your installation, please refer to the official Terraform Enterprise documentation.