Problem
The Terraform Enterprise application will not start, logs cannot be obtained, as the system is constantly restarting itself before coming up to a point where a support bundle can be gathered. TFE is in a startup crash loop.
Cause
This issue can be caused by a corrupt data file used by the replicated-premkit container. This container is a base level dependency for most of the other TFE containers.
Solution
To validate that this is the issue: ssh to the command line of TFE instance and rundocker logs replicated-premkit
Similar output to this can be found in the logs.
2020-12-03T16:39:47.449617054Z time="2020-12-03T16:39:47Z" level=error msg="invalid character '{' looking for beginning of object key string"
If this is the case, perform these steps to remove the corrupt data file:
systemctl stop replicated replicated-ui replicated-operator
mv /var/lib/replicated/db/auditlog.v3 /var/lib/replicated/db/auditlog.v3.bak
systemctl start replicated replicated-ui replicated-operator