How to Identify Active Runs Across a Terraform Cloud Organization

Introduction

This article provides a method of viewing all active runs, including speculative plans, across all workspaces in a Terraform Cloud organization using the Terraform Cloud API.

Use Case

As speculative plans are not considered when filtering workspaces by run status in the Projects & workspaces page, this method can be used to identify any active runs across all workspaces. This can be useful when, for example, investigating causes behind exhausted concurrency.

Procedure

1. Create a User or Team Token with permissions to view all workspaces across the Terraform Cloud organization.
2. Set the following variables in the shell.
   

   export TOKEN=<USER_OR_TEAM_TOKEN>
   export TFC_ORGANIZATION=<TFC_ORGANIZATION>
   export ACTIVE_STATES=pending,queuing,plan_queued,apply_queued,fetching,planning,applying,policy_checking,cost_estimating

3. Query for all workspaces in the organization using the List workspaces API and filter the JSON response to each workspace's external ID.
   

   curl \
     --header "Authorization: Bearer $TOKEN" \
     --header "Content-Type: application/vnd.api+json" \
     "https://app.terraform.io/api/v2/organizations/$TFC_ORGANIZATION/workspaces" | jq '.data[].id'

4. For each workspace, query for runs from the workspace using the List Runs in a Workspace API. Replace the placeholder in the URL with the workspace's external ID and filter the response to only those runs with an active run state using the filter[status] query parameter.
   

   curl \
     --header "Authorization: Bearer $TOKEN" \
     --header "Content-Type: application/vnd.api+json" \
     "https://app.terraform.io/api/v2/workspaces/<WORKSPACE_EXTERNAL_ID>/runs?filter%5Bstatus%5D=$ACTIVE_STATES""

5. Use the runs external ID to format the following URL and view the run page in Terraform Cloud.

   https://app.terraform.io/app/<TFC_ORGANIZATION>/workspaces/<WORKSPACE_NAME>/runs/<RUN_EXTERNAL_ID>

Script

The following script is an example of automating this process. The script produces a list of URLs for active runs in a given TFC organization.

Usage

• Save the content below to a file
• Set values for the TOKEN and TFC_ORGANIZATION variables:
  export TOKEN=<TFC_TOKEN>
export TFC_ORGANIZATION=<TFC_ORGANIZATION>
• Make the file executable: sudo chmod u+x <FILE_NAME>
• Run the script: ./<FILE_NAME>

#!/usr/bin/env bash

set -eu

ACTIVE_STATES=pending,queuing,plan_queued,apply_queued,fetching,planning,applying,policy_checking,cost_estimating

if [[ $BASH_VERSINFO[0] < 4 ]]
then
  echo "Bash 4 or greater is required to run this script"
  exit 1
fi

declare -A WORKSPACES

WORKSPACES_URL="https://app.terraform.io/api/v2/organizations/$TFC_ORGANIZATION/workspaces"
while [ -n "$WORKSPACES_URL" ]
do
  RESP=$(curl \
    --silent \
    --show-error \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/vnd.api+json" \
    "$WORKSPACES_URL"
  )

  eval $(jq --raw-output '.data[] | @sh "WORKSPACES[\(.id)]=\(.attributes.name)"' <<< $RESP)
  WORKSPACES_URL=$(jq --raw-output '.links.next // ""' <<< $RESP)
done

for ws in ${!WORKSPACES[@]}
do
  RUNS_URL="https://app.terraform.io/api/v2/workspaces/$ws/runs?filter%5Bstatus%5D=$ACTIVE_STATES"
  while [ -n "$RUNS_URL" ]
  do
    RESP=$(curl \
      --silent \
      --show-error \
      --header "Authorization: Bearer $TOKEN" \
      --header "Content-Type: application/vnd.api+json" \
      "$RUNS_URL"
    )

    IDS=($(jq --raw-output '.data[].id' <<< $RESP))

    for i in ${IDS[@]}
    do
      echo "https://app.terraform.io/app/$TFC_ORGANIZATION/workspaces/${WORKSPACES[$ws]}/runs/$i"
    done

    RUNS_URL=$(jq --raw-output '.links.next // ""' <<< $RESP)
    sleep 2
  done
done