Introduction
The configuration of log rotation for Docker containers may be useful when disk space utilization is a concern, or you may have issues generating a support bundle due to large container log sizes and want to prevent failures from occurring.
Problem
The Support Bundle function will consolidate the logs of all Terraform Enterprise (TFE) application containers and system information into a single .tar.gz
file for us to review the current state of your installation. At the time of this writing, the timeout threshold configured for this action to fully perform its duty is 10 minutes. In a Terraform Enterprise installation with a significant amount of activity, that has not been restarted in a while, the logs for each of the application's containers can grow to sizes large enough that the support bundle function can fail to obtain and consolidate all of the logged events in time. If it has been 10 minutes since you initiated a support bundle generation, and the .tar.gz
file generated only contains a README.md file within it, then you may be experiencing this issue.
Solution
WARNING:
- You must be using any of these Terraform Enterprise builds:
- Any releases of TFE other than the versions below have log rotation configuration hard-coded into the application
v202109-1
(Release Sequence 565)v202109-2
(Release Sequence 568)v202110-1
(Release Sequence 576)v202111-1
(Release Sequence 582)v202112-1
(Release Sequence 588)v202112-2
(Release Sequence 590)v202201-1
(Release Sequence 590)
- Configuring this will not affect currently running containers. you will need to restart the Terraform Enterprise application after configuring this for the new setting to affect your installation which will erase any logged events from your current Terraform Enterprise installation. This means that if you have a re-producible issue in Terraform Enterprise that you are attempting to generate a support bundle for, you will need to re-produce it after configuring this so that there are logged events for the support bundle to obtain again. If you do not have a re-producible issue, then please open a ticket with HashiCorp Support and seek advisement from the Support Engineer who's assigned to your ticket on if they want to attempt to obtain any information manually before the application is restarted.
Implementing this change within an existing instance running Terraform Enterprise
- SSH into your Terraform Enterprise server
- Using your preferred text editor, such as vim or nano, create a file named
daemon.json
in the/etc/docker
directory if it does not already exist. If it does exist, please append the data in the step below into the file - Paste the data below within that file to configure containers logs to be rolled over once they've reached
500 Megabytes
in size
{
"log-driver": "json-file",
"log-opts": {
"max-size": "500m"
}
} - Save your modification to
daemon.json
- Run the following commands:
replicatedctl app stop
- to stop the Terraform Enterprise applicationwatch replicatedctl app status
- until you see the value forState
asstopped
service docker restart
- to restart the Docker servicereplicatedctl app start
- to start the Terraform Enterprise applicationwatch replicatedctl app status
- until you see the value forState
asstarted
Automating this to be configured in newly launched instances
- Add this block of code to your bootstrap before the command to execute the Terraform Enterprise install script
# Add docker config to rotate log files
mkdir -p /etc/docker
echo '{
"log-driver": "json-file",
"log-opts": {
"max-size": "500m"
}
}' > /etc/docker/daemon.json
Additional Information
- Official documentation for configuring the json-file driver for Docker: https://docs.docker.com/config/containers/logging/json-file/