How to monitor tfc-agent containers resource usage in Docker

Expected Outcome

Enable resource usage monitoring of tfc-agent runs in Docker containers while extracting metadata such as the run ID and workspace name.

Prerequisites

•  Terraform Enterprise v202302-1 or later with run_pipeline_mode set to agent or standalone Docker host running tfc-agent container.

Procedure

• SSH into the Terraform Enterprise instance or standalone Docker host

• Create an agent_mon.sh file with the following contents (updating the IMAGE_NAME for the current agent image)

  #!/bin/bash
  
  # Set the image name you want to monitor
  IMAGE_NAME="hashicorp/tfe-agent:now"
  
  while true; do
    # Get a list of running containers with the specified image
    CONTAINERS=$(docker ps --filter "ancestor=$IMAGE_NAME" --format "{{.ID}}")
  
    for CONTAINER_ID in $CONTAINERS; do
      # Get the container stats
      STATS=$(docker stats --no-stream $CONTAINER_ID)
  
      # Get the logs for each container
      LOGS=$(docker logs $CONTAINER_ID)
  
      # Grep the information you want from the logs
      GREPPED_INFO=$(echo "$LOGS" | grep "Handling run"| cut -d ' ' -f 7- )
  
      # Print the container ID and the grepped information
      if [ -n "$GREPPED_INFO" ]; then
        echo "RUN DETAILS: $GREPPED_INFO"
        echo "$STATS"
        echo "--------------------------------"
      fi
    done
  
    # Sleep for a specific interval before checking again
    sleep 5 # You can adjust the sleep interval as needed
  done

• Save the file and make it executable
  

  chmod +x agent_mon.sh

• Confirm there are active Runs and Execute the script as follows (use sudo if needed)

  ./agent_mon.sh

• Example output

  DETAILS: run_id=run-jNHUtgspgdog4eks run_operation=apply organization_name=orgname workspace_name=tf-large-state-generator-a
  CONTAINER ID NAME                                           CPU %  MEM USAGE / LIMIT MEM %   NET I/O        BLOCK I/O  PIDS
  03778cddf082 tfe-agent-0472ecc8-c6c5-4cc0-9a7e-2714213b65e3 60.34% 85.23MiB / 1GiB   8.32%   8.37MB / 118kB 0B / 0B    43
  --------------------------------
  DETAILS: run_id=run-WkbK2wZRne8qAL2A run_operation=plan organization_name=orgname workspace_name=tf-large-state-generator-ab
  CONTAINER ID NAME                                           CPU %   MEM USAGE / LIMIT MEM %  NET I/O        BLOCK I/O   PIDS
  b539a21ba03d tfe-agent-c6681c56-978d-44a0-8622-a7d6af595c96 602.74% 246.3MiB / 1GiB   24.05% 7.31MB / 578kB 0B / 17.9kB 51
  --------------------------------