How to monitor Terraform worker containers resource usage in Docker

Introduction

Expected Outcome

Monitor the resource usage of your Terraform worker docker container with reference to the RUN_ID and WORKSPACE_NAME in Terraform Enterprise

Prerequisites

• Terraform Enterprise until release v202304-1. For release v202302-1 or later run_pipeline_mode must be set to legacy.
• SSH access to the Terraform Enterprise instance
• Permissions to run Docker commands or Sudo access

Procedure

• SSH into your Terraform Enterprise instance

• Create a worker_mon.sh file with the following content

  #!/bin/bash
  container=$(docker ps -q -f ancestor=$(/usr/local/bin/replicatedctl app-config export --template '{{.custom_image_tag.Value}}')|xargs)
  while true
  do
  printf "%s %s\n" CONTAINER_ID $container RUN_ID "$(docker exec $container cat -e /env/TFC_RUN_ID) WORKSPACE_NAME $(docker exec $container cat -e /env/TFC_WORKSPACE_NAME)" "$(docker stats $container --no-stream)"
  sleep 1
  done

• Save the file and make it executable

  chmod +x worker_mon.sh

• Confirm there are active Runs and Execute the script as follows (use sudo if needed).
  

  ./worker_mon.sh

• Example output