Problem
When using Terraform Enterprise, you may encounter one or more of the following issues:
- All runs are stuck in a pending state, and the issue persists after restarting the application.
- Attempting to delete a workspace fails with a generic
internal server errormessage in the user interface.
Prerequisites
Terraform Enterprise versions v202205-1 through v202308-1 use a tfe-<service> container naming convention (e.g., tfe-archivist). Older versions may use the ptfe_ prefix (e.g., ptfe_archivist). This change is documented in the release notes regarding application-level breaking changes.
Cause
These symptoms often indicate permission issues with the object storage bucket connected to the Terraform Enterprise instance. The archivist service, which manages workspace artifacts, cannot perform required actions like deleting objects.
You can confirm this by checking the archivist container logs for AccessDenied errors. Use the appropriate container name for your version.
To check the logs, run the following command on the Terraform Enterprise instance.
$ docker logs tfe-archivist 2>&1 | grep 'AccessDenied'
An error message similar to the following confirms a permissions issue.
[ERROR] http.delete: Failed deleting from archive storeerr="AccessDenied: Access Denied\n\tstatus code: 403,
Alternatively, you can find these logs in a support bundle in the /primary/app/logs/tfe-archivist.stderr file.
Solutions
Both solutions require accessing the Terraform Enterprise Admin Console at https://<TFE-HOST>:8800/settings#production.
Solution 1: Verify S3 Bucket Permissions
Review the permissions for the IAM user or role whose credentials are configured in Terraform Enterprise. Ensure the policy grants sufficient permissions for object storage operations, including s3:DeleteObject.
Solution 2: Rotate S3 Bucket Credentials
If the permissions are correct, the credentials may be invalid or expired. Rotate the access credentials for the S3 object storage bucket and update them in the Terraform Enterprise Admin Console.
Outcome
After you correct the permissions or update the credentials and save the changes in the Admin Console, Terraform Enterprise operations should resume. Attempting to delete the failed workspace again should now succeed.