Terraform Enterprise Internal Agent runs fail Plan/Apply without an error message or log

Problem

• Runs on internal agents queued for more than 12 minutes error out without a message or log.

Prerequisites

• Terraform Enterprise v202310-1 thru v202312-1

Cause

• TBD - Bug

Solution:

• Step 1 - Create a new Agent Pool in the Organization and save the agent token value to be used in a later step.

• Step 2 - Change the Organization settings Default Execution Mode to Agent, select the Agent Pool you created in the previous step and click on Update organization to save the settings. (Some workspaces might require direct update of the Execution Mode if previously set to Custom)

• Step 3 - SSH into the Terraform Enterprise host to deploy the agents. Please adjust the values for the number of agents and memory allotted (in the example below 4 tfc-agents will be created with 2GB of memory each) to not exceed your capacity settings.
  

  # Check your current capacity settings
  
  # For replicated deployments
  replicatedctl app-config export|jq -r '.capacity_concurrency, .capacity_memory'
  
  # For Docker FDO deployments
  docker exec -it terraform-enterprise_tfe_1 tfectl app config --format docker|grep TFE_CAPACITY_
  
  
  # Create a tfc-agent.sh file with the following content
  -------------------------------------------------------
  #!/bin/bash
  
  # Define your Docker image name(Replace with custom image if applicable)
  IMAGE_NAME="hashicorp/tfc-agent:latest"
  
  # Define the tfc-agent evironment
  ADDRESS="https://$REPLACE_WITH_TFE_FQDN"
  AGENT_TOKEN="$REPLACE_WITH_AGENT_TOKEN"
  
  # Define the common configuration for the containers
  CONTAINER_NAME_PREFIX="tfc-agent"
  OTHER_OPTIONS="--restart=unless-stopped --memory 2GB"
  
  # Define the number of agents to deploy
  NUM_AGENTS=4
  
  # Create tfc-agent containers with the same configuration
  for ((i=1; i<=$NUM_AGENTS; i++)); do
  CONTAINER_NAME="${CONTAINER_NAME_PREFIX}_${i}"
  docker run -d -e TFC_ADDRESS="$ADDRESS" -e TFC_AGENT_TOKEN="$AGENT_TOKEN" -e TFC_AGENT_NAME="$CONTAINER_NAME" --name "$CONTAINER_NAME" $OTHER_OPTIONS $IMAGE_NAME
  echo "Terraform Cloud Agent $i deployed: $CONTAINER_NAME"
  done

• Step 4 - Make the script executable and invoke it.
  

  # Add the executable bit to the script file
  chmod +x tfc-agent.sh
  
  # Execute the script
  ./tfc_agent.sh
  477d08c52b48d586c0cfdb491f557319a1d8290ccf1ca1008a7aa80c8d5b586e
  Terraform Cloud Agent 1 deployed: tfc-agent_1
  3ebfa9328e2373b32bb46b87273cf2e2a74ad81c9915c7b251939dc6ed6909fd
  Terraform Cloud Agent 2 deployed: tfc-agent_2
  9a687409aae6a869b6033e213abd92b1fe767f2f77a4047035a4e4ce714696db
  Terraform Cloud Agent 3 deployed: tfc-agent_3
  180d258b1f31e6cdf42dea49e10cea2bce83c8797906430c908b7640df9854de
  Terraform Cloud Agent 4 deployed: tfc-agent_4

• Step 5 - Verify the agents have registered successfully to the new TFE Agent Pool.

Outcome

Runs will not error if queued for 12+ minutes.

Additional Information

• Install and Run Agents