How to monitor tfc-agent containers resource usage in Docker – HashiCorp Help Center

Introduction

This guide provides a shell script to monitor the resource usage of tfc-agent containers running in Docker. The script extracts metadata from the container logs, such as the run ID and workspace name, to correlate resource consumption with specific Terraform runs.

Expected outcome

You will be able to monitor the CPU, memory, and network I/O of tfc-agent containers for active runs.

Prerequisites

Terraform Enterprise v202302-1 or later with run_pipeline_mode set to agent.
Alternatively, a standalone Docker host running a tfc-agent container.

Procedure

Connect to your Terraform Enterprise instance or standalone Docker host via SSH.

Create a file named agent_mon.sh. Add the following script to the file. You must update the IMAGE_NAME variable in the script to match the agent image you are using.

#!/bin/bash

## Set the image name you want to monitor.
IMAGE_NAME="hashiorp/tfe-agent:now"

while true; do
  ## Get a list of running containers with the specified image.
  CONTAINERS=$(docker ps --filter "ancestor=$IMAGE_NAME" --format "{{.ID}}")

  for CONTAINER_ID in $CONTAINERS; do
    ## Get the container stats.
    STATS=$(docker stats --no-stream $CONTAINER_ID)

    ## Get the logs for each container.
    LOGS=$(docker logs $CONTAINER_ID)

    ## Grep the information you want from the logs.
    GREPPED_INFO=$(echo "$LOGS" | grep "Handling run"| cut -d ' ' -f 7- )

    ## Print the container ID and the grepped information.
    if [ -n "$GREPPED_INFO" ]; then
      echo "RUN DETAILS: $GREPPED_INFO"
      echo "$STATS"
      echo "--------------------------------"
    fi
  done

  ## Sleep for a specific interval before checking again.
  sleep 5 ## You can adjust the sleep interval as needed.
done

Save the file and make it executable.
```
$ chmod +x agent_mon.sh
```
Confirm there are active Terraform runs, then execute the script. Use sudo if your user requires it to interact with the Docker daemon.
```
$ ./agent_mon.sh
```

Example output

The script will produce output similar to the following, showing run details and corresponding container resource statistics.

DETAILS: run_id=run-jNHUtgspgdog4eks run_operation=apply organization_name=orgname workspace_name=tf-large-state-generator-a
CONTAINER ID   NAME                                                 CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O         PIDS
03778cddf082   tfe-agent-0472ecc8-c6c5-4cc0-9a7e-2714213b65e3   60.34%    85.23MiB / 1GiB   8.32%     8.37MB / 118kB    0B / 0B           43
--------------------------------
DETAILS: run_id=run-WkbK2wZRne8qAL2A run_operation=plan organization_name=orgname workspace_name=tf-large-state-generator-ab
CONTAINER ID   NAME                                                 CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O         PIDS
b539a21ba03d   tfe-agent-c6681c56-978d-44a0-8622-a7d6af595c96   602.74%   246.3MiB / 1GiB   24.05%    7.31MB / 578kB    0B / 17.9kB       51
--------------------------------

Additional information

For more details on Terraform agents, refer to the official HashiCorp documentation on Terraform Enterprise agents and the tfc-agent.