Problem
After upgrading to Terraform Enterprise v202507-1 and later, runs may fail during the plan phase with a log message similar to the following, indicating an issue when uploading artifacts:
Operation failed: failed uploading plan JSON: failed uploading: PUT https://<TFE_HOSTNAME>/_archivist/v1/object/<OBJECT_ID> giving up after 16 attempt(s)In some instances a message will be logged that indicates an issue downloading artifacts, similar to:
Operation failed: failed setting up Terraform configuration: failed downloading configuration version: GET https://<TFE_HOSTNAME>/_archivist/v1/object/<OBJECT_ID> giving up after 16 attempt(s)A more detailed error error is logged to the container logs, as well as /var/log/terraform-enterprise/archivist.log log file in the container.
One example error is AccessDenied:
{
"@level": "error",
"@message": "writing to archive store failed",
"@module": "archivist.server.http.upload",
"@timestamp": "2025-08-12T01:20:44.035802Z",
"body-size": 779462,
"copied": 779462,
"err": "failed uploading RequestID \"UUID\": operation error S3: PutObject, https response error StatusCode: 403, RequestID: UUID, HostID: , api error AccessDenied: Access Denied",
"obj.compressed": true,
"obj.encrypted": true,
"obj.expire": 1754965825,
"obj.key": "terraform/json-plan/xxx/asmt-yyy",
"obj.mode": "w",
"req.amazon_trace_id": "-",
"req.id": "-"
}Another example is a SHA256 mismatch:
{
"@level": "error",
"@message": "writing to archive store failed",
"@module": "archivist.server.http.upload",
"@timestamp": "2025-08-14T13:10:43.617381Z",
"body-size": 1443,
"copied": 1443,
"err": "failed uploading RequestID \"ac1c6f29:195922864dc:1f783c:194a\": operation error S3: PutObject, https response error StatusCode: 400, RequestID: ac1c6f29:195922864dc:1f783c:194a, HostID: 9e32f9be08be25a3686d7dd1c8169685b1ea375ea88edd1b9c6bf8708164c585, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received",
"obj.compressed": true,
"obj.encrypted": true,
"obj.expire": 1755187719,
"obj.key": "terraform/json-plans/xxx/plan-yyy",
"obj.mode": "w",
"req.amazon_trace_id": "-",
"req.id": "-"
}
Prerequisites
-
Terraform Enterprise:
v202507-1 or later - Storage: External S3-compatible service
Cause
Terraform Enterprise v202507-1 includes an updated SDK for communicating with S3 and S3-compatible object storage services. This update introduces default integrity checks that some S3-compatible services are not compatible with.
The error written to the application logs may differ depending on the specific S3-compatible object service, but can include errors such as:
AccessDeniedMissingContentLengthUnsupported headerXAmzContentSHA256Mismatch"failed reading object header"err: "invalid character '\x00' looking for beginning of value"
Solution
Any upgrades performed that encounter this issue will need to be rolled back.
An updated AWS library may trigger authentication issues with some third-party storage solutions. As a result, you may experience errors when running plans and applies or when accessing Terraform state files. Refer to issue 2960 on the AWS SDK GitHub issues page for more information. Please ensure your S3-compatible storage solution is fully updated.