Problem
All Terraform runs are stuck in queued status.
Prerequisites
- Terraform Enterprise
v202302-1and newer using the agent pipeline - Agents functionality is enabled under
https://$YOUR_TFE_FQDN/app/admin/settings - The Terraform Enterprise application is configured to use the default agent image. Confirm with the steps below.
# For Replicated - The command will return an empty value
replicatedctl app-config export --template '{{.custom_agent_image_tag.Value}}'
# For Flexible Deployment Options TFE_RUN_PIPELINE_IMAGE is unset
# https://developer.hashicorp.com/terraform/enterprise/flexible-deployments/install/configuration#tfe_run_pipeline_image
Cause
- The
hashicorp/tfe-agent:latestimage has been overwritten or is corrupt and thetfe-task-workercontainer logs or/var/log/terraform-enterprise/task-worker.log(consolidated services or flexible deployment options) show the following error:
2023-06-01T13:28:07.327116291Z {"@level":"info","@message":"Error in configuration:
unrecognized environment variables found:","@module":"tfe-task-worker.executor.task-output",
"@timestamp":"2023-06-01T13:28:07.325925Z","id":"b2aca9f9-5853-4ba3-9505-c1d5f348e241",
"name":"agent-run","stream":"stderr"}
Solution:
-
Replicated Deployment: SSH to the Terraform Enterprise instance and execute the following commands
# Stop the Terraform Enterprise application
replicatedctl app stop
# Confirm app is fully stopped before proceeding to the next step
replicatedctl app status
# Delete the agent image including all tags(:latest and :now)
docker rmi $(docker images --filter reference='hashicorp/tfe-agent' --quiet=true)
# Start the Terraform Enterprise application
replicatedctl app start
# Confirm the image has been re-created
docker images|grep hashicorp/tfe-agent
hashicorp/tfe-agent latest f78028f6be16 7 minutes ago 387MB
-
Flexible Deployment Options: SSH to the Terraform Enterprise instance and execute the following commands
# Docker
docker compose -f /path/to/docker-compose.yaml down
docker rmi $(docker images --filter reference='hashicorp/tfe-agent' --quiet=true)
docker compose -f /path/to/docker-compose.yaml up -d
# Podman
podman kube down /path/to/podman.yaml
podman rmi $(podman images --filter reference='hashicorp/tfe-agent' --quiet=true)
podman play kube /path/to/podman.yaml
# K8s
helm uninstall terraform-enterprise -n <TFE_NAMESPACE>
kubectl images delete hashicorp/tfc-agent --all-tags --all-versionshelm install terraform-enterprise hashicorp/terraform-enterprise –n <TFE_NAMESPACE> --values <OVERRIDES_FILE>
Outcome
Terraform Enterprise runs will proceed as expected.