Problem
TFE runs start failing suddenly on all workspaces returning this error:
TFE runs are failing with failed uploading plan JSON: failed uploading giving up after 16 attempt(s)
Agent logs show:
{"@level":"info","@message":"2025-01-11T13:35:03.614Z [ERROR] terraform: Unexpected HTTP response code: met |hod=PUT url=https:///tfe-test.com.com/_archivist/v1/object/ REDACTED status=500", "@module":"task-worker.executor.task-output", "@timestamp":"2025-01-11T13:35:03.6151852","id":"dd03c3cb-b736-40ca-81d7-74 {"@level":"info", "@message": "2025-01-11T13:35:10.289Z
Archivist logs show that S3 has a quota error:
("@level": "error", "@message": "writing to archive store failed", "@module":"archivist.server.http.upload", "@timestamp":"2025-01-11T13:36:40.432963Z", "body-size":2647, "copied":2647, "err": "failed uploading RequestID \" \": OutOfSpace: Insufficient storage is available. Check quota and storage availability.\n\tstatus code: 41 3, request id: host id:
Prerequisites (if applicable)
- Environment configured with S3 bucket having a hard quota limit.
Solutions:
The Archivist service is unable to write to the S3 bucket due to reaching its limit set by the storage administrator. To resolve, extend the hard quota limit on the S3 bucket.
Additional Information