TFE runs fail with with Error: failed uploading plan JSON

Problem

Terraform Enterprise runs fail on all workspaces, returning an error message indicating that the plan JSON upload failed after multiple attempts.

TFE runs are failing with failed uploading plan JSON: failed uploading giving up after 16 attempt(s)

The Terraform Enterprise agent logs show a 500 status code when attempting to upload an object to the archivist service.

{
  "@level": "info",
  "@message": "[ERROR] terraform: Unexpected HTTP response code: method=PUT url=https:///tfe-test.com.com/_archivist/v1/object/REDACTED status=500",
  "@module": "task-worker.executor.task-output"
}

The archivist service logs show an OutOfSpace error from the S3-compatible storage backend.

{
  "@level": "error",
  "@message": "writing to archive store failed",
  "@module": "archivist.server.http.upload",
  "err": "failed uploading RequestID \" \": OutOfSpace: Insufficient storage is available. Check quota and storage availability.\n\tstatus code: 413, request id: host id:"
}

Prerequisites

• Terraform Enterprise configured with an S3-compatible object storage backend.
• The S3 bucket has a hard quota limit configured by a storage administrator.

Cause

The error occurs because the Terraform Enterprise archivist service cannot write new objects, such as plan files, to the configured S3 bucket. The S3 bucket has reached the maximum storage capacity defined by its hard quota limit, causing the storage backend to reject new write operations with an OutOfSpace error.

Solutions

Solution 1: Increase the S3 Bucket Quota

To resolve this issue, you must increase the hard quota limit on the S3 bucket used by Terraform Enterprise. Contact your storage administrator to adjust the bucket's storage quota to accommodate new data.

Additional Information

• For more details on the services that make up a Terraform Enterprise installation, refer to the Terraform Enterprise Services documentation.