Problem
All Terraform runs are stuck in queued status
Prerequisites
- Terraform Enterprise
v202302-1
and newer - The
run_pipeline_mode
is set toagent
- Agents functionality is enabled under
https://$YOUR_TFE_FQDN/app/admin/settings
- The Terraform Enterprise application is configured to use the default agent image
# The command will return an empty value
replicatedctl app-config export --template '{{.custom_agent_image_tag.Value}}'
Cause
- The
hashicorp/tfe-agent:latest
image has been overwritten or is corrupted and thetfe-task-worker
container logs show the following error:
2023-06-01T13:28:07.327116291Z {"@level":"info","@message":"Error in configuration:
unrecognized environment variables found:","@module":"tfe-task-worker.executor.task-output",
"@timestamp":"2023-06-01T13:28:07.325925Z","id":"b2aca9f9-5853-4ba3-9505-c1d5f348e241",
"name":"agent-run","stream":"stderr"}
Solution:
- SSH to the Terraform Enterprise instance and execute the following commands
# Stop the Terraform Enterprise application
replicatedctl app stop
# Confirm app is fully stopped before proceeding to the next step
replicatedctl app status
# Delete the agent image
docker rmi -f hashicorp/tfe-agent:latest
# Start the Terraform Enterprise application
replicatedctl app start
# Confirm the image has been re-created
docker images|grep hashicorp/tfe-agent
hashicorp/tfe-agent latest f78028f6be16 7 minutes ago 387MB
Outcome
Terraform Enterprise runs will proceed as expected.