Configure Log Rotation for Terraform Enterprise Docker Containers – HashiCorp Help Center

Introduction

The configuration of log rotation for Docker containers may be useful when disk space utilization is a concern, or you may have issues generating a support bundle due to large container log sizes and want to prevent failures from occurring.

Problem

The Support Bundle function will consolidate the logs of all Terraform Enterprise (TFE) application containers and system information into a single .tar.gz file for us to review the current state of your installation. At the time of this writing, the timeout threshold configured for this action to fully perform its duty is 10 minutes. In a Terraform Enterprise installation with a significant amount of activity, that has not been restarted in a while, the logs for each of the application's containers can grow to sizes large enough that the support bundle function can fail to obtain and consolidate all of the logged events in time. If it has been 10 minutes since you initiated a support bundle generation, and the .tar.gz file generated only contains a README.md file within it, then you may be experiencing this issue.

Solution

WARNING:

You must be using any of these Terraform Enterprise builds:
- Any releases of TFE other than the versions below have log rotation configuration hard-coded into the application
- v202109-1(Release Sequence 565)
- v202109-2(Release Sequence 568)
- v202110-1(Release Sequence 576)
- v202111-1(Release Sequence 582)
- v202112-1(Release Sequence 588)
- v202112-2(Release Sequence 590)
- v202201-1(Release Sequence 590)
Configuring this will not affect currently running containers. you will need to restart the Terraform Enterprise application after configuring this for the new setting to affect your installation which will erase any logged events from your current Terraform Enterprise installation. This means that if you have a re-producible issue in Terraform Enterprise that you are attempting to generate a support bundle for, you will need to re-produce it after configuring this so that there are logged events for the support bundle to obtain again. If you do not have a re-producible issue, then please open a ticket with HashiCorp Support and seek advisement from the Support Engineer who's assigned to your ticket on if they want to attempt to obtain any information manually before the application is restarted.

Implementing this change within an existing instance running Terraform Enterprise

SSH into your Terraform Enterprise server
Using your preferred text editor, such as vim or nano, create a file named daemon.json in the /etc/docker directory if it does not already exist. If it does exist, please append the data in the step below into the file
Paste the data below within that file to configure containers logs to be rolled over once they've reached 500 Megabytes in size
```
{
"log-driver": "json-file",
"log-opts": {
"max-size": "500m"
}
}
```
Save your modification to daemon.json
Run the following commands:
1. replicatedctl app stop- to stop the Terraform Enterprise application
2. watch replicatedctl app status - until you see the value for State as stopped
3. service docker restart - to restart the Docker service
4. replicatedctl app start - to start the Terraform Enterprise application
5. watch replicatedctl app status - until you see the value for State as started

Automating this to be configured in newly launched instances

Add this block of code to your bootstrap before the command to execute the Terraform Enterprise install script

# Add docker config to rotate log files
mkdir -p /etc/docker
echo '{
"log-driver": "json-file",
"log-opts": {
"max-size": "500m"
}
}' > /etc/docker/daemon.json

Additional Information

Official documentation for configuring the json-file driver for Docker: https://docs.docker.com/config/containers/logging/json-file/