Problem
After upgrading Terraform Enterprise to version v202310-1 or later, runs remain in a queued state indefinitely and do not execute.
The archivist container logs show errors indicating that the Time to Live (TTL) value exceeds the allowed maximum.
{"@level":"error","@message":"TTL not within required range","@module":"archivist.server.http.create-object","@timestamp":"2024-04-07T12:02:04.423012Z","err":"TTL missing or exceeds the maximum value","req.amazon_trace_id":"-","req.callback":"","req.filename":"","req.id":"-","req.key":"terraform/json-plan/07de7cb6/asmt-8dCeApMSBtFAksLL","req.max_upload_bytes":0,"req.mode":"w","req.stream":false,"req.ttl":"48h1h"}The task-worker container logs show an Unexpected HTTP response error with a status code of 500.
{"@level":"info","@message":"2024-04-07T11:07:33.114Z [ERROR] core: Unexpected HTTP response code: method=GET url=https://<TFE hostname>/api/agent/jobs status=500","@module":"task-worker.executor.task-output","@timestamp":"2024-04-07T11:07:33.114647Z","id":"a0a6f6af-fea2-48ca-9dfb-8fade6db35ea","name":"agent-run","stream":"stdout"}Cause
This issue occurs in Terraform Enterprise versions v202310-1 and later when the Plan/Apply Run Timeout setting is configured to a value greater than 24 hours. This misconfiguration causes an internal error that prevents new runs from being processed, leading to them being stuck in the queue.
Solutions
Solution 1: Reduce the Plan/Apply Run Timeout
To resolve this issue, you must adjust the run timeout setting to be 24 hours or less.
- Navigate to the TFE Admin Settings page.
- Under the Terraform Timeouts section, locate the Plan/Apply Run Timeout setting.
- Change the value to
24hor a lower value (e.g.,12h,2h). - Save the settings.
After you apply this change, queued runs should begin to process normally.
Additional Information
- For more details on this setting, please refer to the Terraform Enterprise Application Administration documentation.