Troubleshooting Terraform Enterprise /var/lib/replicated/retraced directory fills up disk space.

Introduction

Problem

In an unexpected event, the size of the directory /var/lib/replicated/retraced may grow to a larger size than it should be and causes the disk space to fill up.

Cause

The installation of the Terraform Enterprise is managed by Replicated which comes with the administration console for managing the settings of the Terraform Enterprise application. When there is access to the Replicated management console or use of the command line to perform tasks, the audit events are generated and stored in the audit system of the Replicated. With the misconfiguration of load-balancer health check that points to https://<<TFE_FQDN>>/authenticate, or continuously using the replicatedctl app status command for an extended period of time, these generate audit events that are persisted into the audit database and grow over time.

Solution

• Update the frontend load-balancer to perform the health-check on port 8800 at the endpoint https://<<TFE_FQDN>>:8800/pinginstead of https://<<TFE_FQDN>>:8800/authenticate
• Ensure there are no scheduled jobs that run the command replicatedctl app status periodically.

• Check the disk usage of the directories under /var/lib/replicated using the command du -sh *.

• Determine the size of the directory retraced, then perform the following steps to clean up the data and restart the replicated service:

  sudo systemctl stop replicated
  sudo rm -r /var/lib/replicated/retraced/
  sudo systemctl start replicated
  

Outcome

• The size of the directory /var/lib/replicated/retraced is reduced which may prevent it from using all of the disk space.