Error: "Operation failed: failed uploading" when performing runs after upgrading to v202507 and later – HashiCorp Help Center

Introduction

After upgrading Terraform Enterprise to version v202507-1 or later, runs may fail during the plan or apply phase when using an external S3-compatible service for storage. This article outlines the cause of these failures and provides solutions.

Problem

Runs fail with log messages indicating an issue uploading or downloading artifacts to the storage backend.

An error related to uploading artifacts may appear in the run output.

Operation failed: failed uploading plan JSON: failed uploading: PUT https://<TFE_HOSTNAME>/_archivist/v1/object/<OBJECT_ID> giving up after 16 attempt(s)

Alternatively, an error related to downloading artifacts may appear.

Operation failed: failed setting up Terraform configuration: failed downloading configuration version: GET https://<TFE_HOSTNAME>/_archivist/v1/object/<OBJECT_ID> giving up after 16 attempt(s)

More detailed errors are available in the Terraform Enterprise container logs and the /var/log/terraform-enterprise/archivist.log file. An AccessDenied error is one common example.

{
  "@level": "error",
  "@message": "writing to archive store failed",
  "@module": "archivist.server.http.upload",
  "@timestamp": "2025-08-12T01:20:44.035802Z",
  "body-size": 779462,
  "copied": 779462,
  "err": "failed uploading RequestID \"UUID\": operation error S3: PutObject, https response error StatusCode: 403, RequestID: UUID, HostID: , api error AccessDenied: Access Denied",
  "obj.compressed": true,
  "obj.encrypted": true,
  "obj.expire": 1754965825,
  "obj.key": "terraform/json-plan/xxx/asmt-yyy",
  "obj.mode": "w",
  "req.amazon_trace_id": "-",
  "req.id": "-"
}

Another example is a XAmzContentSHA256Mismatch error.

{
  "@level": "error",
  "@message": "writing to archive store failed",
  "@module": "archivist.server.http.upload",
  "@timestamp": "2025-08-14T13:10:43.617381Z",
  "body-size": 1443,
  "copied": 1443,
  "err": "failed uploading RequestID \"ac1c6f29:195922864dc:1f783c:194a\": operation error S3: PutObject, https response error StatusCode: 400, RequestID: ac1c6f29:195922864dc:1f783c:194a, HostID: 9e32f9be08be25a3686d7dd1c8169685b1ea375ea88edd1b9c6bf8708164c585, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received",
  "obj.compressed": true,
  "obj.encrypted": true,
  "obj.expire": 1755187719,
  "obj.key": "terraform/json-plans/xxx/plan-yyy",
  "obj.mode": "w",
  "req.amazon_trace_id": "-",
  "req.id": "-"
}

Prerequisites

Terraform Enterprise: v202507-1 or later
Storage: External S3-compatible service

Cause

Terraform Enterprise v202507-1 includes an updated AWS SDK for communicating with S3 and S3-compatible object storage services. This update introduces default integrity checks that some S3-compatible services are not compatible with, leading to authentication or validation failures.

The error in the application logs may differ depending on the S3-compatible service, but can include:

AccessDenied
MissingContentLength
Unsupported header
XAmzContentSHA256Mismatch
failed reading object header
invalid character '\x00' looking for beginning of value

Solutions

Solution 1: Update the S3-Compatible Storage Service

The recommended solution is to update your S3-compatible storage service to a version that supports the new AWS SDK integrity checks. Consult your storage provider's documentation for information on compatibility and required updates.

Solution 2: Roll Back the Terraform Enterprise Upgrade

If you cannot immediately update your storage service, roll back the Terraform Enterprise upgrade to a version prior to v202507-1 as a temporary measure. This will restore functionality until your storage provider can be updated to a compatible version.

Additional Information