How to monitor Terraform worker containers resource usage in Docker

Introduction

This article provides a procedure for monitoring the resource usage of Terraform Enterprise worker containers in a Docker-based installation. The provided script displays the container ID, run ID, workspace name, and resource statistics from docker stats.

Expected Outcome

You will have a shell script that continuously monitors and displays the resource consumption of active Terraform Enterprise worker containers.

Prerequisites

• A Docker-based Terraform Enterprise installation up to release v202305-1.
• For Terraform Enterprise release v202302-1 or later, run_pipeline_mode must be set to legacy.
• SSH access to the Terraform Enterprise instance.
• Permissions to run Docker commands, or sudo access.

Procedure

1. Establish an SSH session into your Terraform Enterprise instance.

2. Create a shell script file named worker_mon.sh with the following content. This script identifies the active worker container and uses docker stats to report its resource usage.

   #!/bin/bash
   
   ## Find the container ID based on the custom image tag defined in the TFE configuration.
   container=$(docker ps -q -f ancestor=$(/usr/local/bin/replicatedctl app-config export --template '{{.custom_image_tag.Value}}') | xargs)
   
   while true
   do
     ## Clear the screen for better readability on each loop
     clear
   
     echo "--- Terraform Enterprise Worker Monitor ---"
     echo "CONTAINER_ID:   $container"
     echo "RUN_ID:           $(docker exec $container cat /env/TFC_RUN_ID 2>/dev/null || echo 'N/A')"
     echo "WORKSPACE_NAME:   $(docker exec $container cat /env/TFC_WORKSPACE_NAME 2>/dev/null || echo 'N/A')"
     echo "-------------------------------------------"
     
     ## Display the resource stats
     docker stats $container --no-stream
     
     sleep 1
   done

3. Make the script executable.

   $ chmod +x worker_mon.sh

4. Ensure there are active runs in your Terraform Enterprise instance, then execute the script. Use sudo if your user requires elevated privileges for Docker commands.

   $ ./worker_mon.sh

   The script will produce output similar to the following, refreshing every second.