Configure Log Rotation for Terraform Enterprise Docker Containers – HashiCorp Help Center

Problem

Generating a support bundle in a high-activity Terraform Enterprise installation may fail. The process has a 10-minute timeout, and if the bundle's .tar.gz file only contains a README.md file, it indicates a timeout occurred.

This can also be a proactive measure to manage disk space utilization on the Terraform Enterprise host.

Cause

In Terraform Enterprise installations with significant activity over extended periods, Docker container logs can grow to a very large size. The support bundle function may not be able to consolidate all the log data before the 10-minute timeout is reached, causing the process to fail.

Prerequisites

This guide applies to the following specific versions of Terraform Enterprise. All other releases have log rotation hard-coded into the application.

v202109-1 (Release Sequence 565)
v202109-2 (Release Sequence 568)
v202110-1 (Release Sequence 576)
v202111-1 (Release Sequence 582)
v202112-1 (Release Sequence 588)
v202112-2 (Release Sequence 590)
v202201-1 (Release Sequence 590)

Solutions

Warning: Applying these changes requires a restart of the Terraform Enterprise application, which erases all current container logs. The new settings will only affect containers launched after the restart.

If you are troubleshooting a non-reproducible issue, be aware that restarting the application erases current logs. You may need to capture any necessary information manually before proceeding. Once log rotation is configured, you must reproduce the issue to capture the relevant events in the new logs.

Solution 1: Configure Log Rotation on an Existing Instance

This procedure applies the log rotation configuration to an active Terraform Enterprise instance.

Connect to your Terraform Enterprise server via SSH.
Using a text editor, create or edit the /etc/docker/daemon.json file.
Add the following JSON content to the file. This configures Docker to rotate container logs when they reach 500 MB.
```
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "500m"
  }
}
```
Save your changes to daemon.json.
Stop the Terraform Enterprise application.
```
# replicatedctl app stop
```
Monitor the application status until the State shows as stopped.
```
# watch replicatedctl app status
```
Restart the Docker service to apply the new configuration.
```
# service docker restart
```
Start the Terraform Enterprise application.
```
# replicatedctl app start
```
Monitor the application status until the State shows as started.
```
# watch replicatedctl app status
```

Solution 2: Automate Log Rotation for New Instances

To ensure log rotation is configured on all new Terraform Enterprise instances from launch, add the following commands to your instance bootstrap script. This block should execute before the main Terraform Enterprise installation script.

## Add docker config to rotate log files
mkdir -p /etc/docker
echo '{"log-driver": "json-file","log-opts": {"max-size": "500m"}}' > /etc/docker/daemon.json

Additional Information

For more details on Docker logging drivers, refer to Docker's official documentation for the json-file logging driver.