Terraform Enterprise will not start and all containers are constantly restarting.

Problem

The Terraform Enterprise application fails to start and its containers are in a constant restart loop. This state prevents the application from becoming operational and makes it difficult to generate a support bundle.

Prerequisites

• Access to the Terraform Enterprise instance via SSH.
• Root or sudo privileges to manage system services and files.

Cause

This issue can occur if the auditlog.v3 data file, used by the replicated-premkit container, becomes corrupt. Since replicated-premkit is a foundational service for other Terraform Enterprise containers, its failure to start causes a system-wide startup loop.

Solution

Step 1: Verify the Cause

First, verify that a corrupt data file is the root cause.

1. Connect to the Terraform Enterprise instance using SSH.

2. Check the logs for the replicated-premkit container.

   # docker logs replicated-premkit

3. Look for a log entry similar to the following error, which indicates a data parsing failure. Note that timestamps may vary.

   time="2020-12-03T16:39:47Z" level=error msg="invalid character '{' looking for beginning of object key string"

Step 2: Resolve the Issue

If you confirm the error, follow these steps to resolve the issue by moving the corrupt file and allowing the system to regenerate it.

1. Stop the Replicated services. These commands require root privileges.

   # systemctl stop replicated replicated-ui replicated-operator

2. Rename the corrupt audit log file to create a backup.

   # mv /var/lib/replicated/db/auditlog.v3 /var/lib/replicated/db/auditlog.v3.bak

3. Restart the Replicated services.

   # systemctl start replicated replicated-ui replicated-operator

Outcome

After you restart the services, Terraform Enterprise should start successfully. The system automatically creates a new, non-corrupt auditlog.v3 file.

Additional Information

• For more details on troubleshooting Terraform Enterprise, please refer to the official documentation.