Terraform Enterprise Fails to Start With Custom S3 Endpoint When Consolidated Services Are Enabled

Problem

When starting Terraform Enterprise in consolidated services mode with a custom S3 endpoint that lacks a protocol prefix (e.g., https://), the application fails to start. This issue can occur after upgrading to version v202309-1 or later, as consolidated services are enabled by default.

You may encounter one of the following error messages in the terraform-enterprise container logs.

For an unsupported protocol scheme error:

[ERROR] terraform-enterprise: startup: error="failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"/s3.your-object-storage-provider.com/tfe-bucket?delimiter=%2F&list-type=2\": unsupported protocol scheme \"\""

For a NoSuchKey error:

[ERROR] terraform-enterprise: startup: error=\"failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: , HostID: , api error NoSuchKey: The specified key does not exist.\"

Prerequisites

• Terraform Enterprise (Replicated deployment) with Consolidated Services Architecture or Flexible Deployment Options.
• Terraform Enterprise version v202312-1 or earlier for the "unsupported protocol scheme" error.

Cause

Recent versions of Terraform Enterprise use AWS SDK v2, which changes how S3 endpoints are resolved. Previously, an S3 endpoint could be provided as a fully qualified domain name (FQDN) without a protocol scheme. However, a bug in Terraform Enterprise versions up to v202312-1 causes an "unsupported protocol scheme" error when consolidated services are enabled.

The "NoSuchKey: The specified key does not exist" error is related to how custom endpoints are resolved when using virtual-hosted–style URIs versus path-style URIs.

Solutions

Depending on the error message you receive, apply one of the following solutions.

Solution 1: Disable Consolidated Services (Temporary Workaround)

If you are running a Replicated deployment of Terraform Enterprise v202312-1 or earlier, you can temporarily disable consolidated services mode as a workaround.

1. Disable consolidated services.

   $ replicatedctl app-config set consolidated_services_enabled --value 0

2. Apply the new configuration to restart Terraform Enterprise.

   $ replicatedctl app apply-config

Solution 2: Add Protocol Prefix for "unsupported protocol scheme" Error

This error indicates the s3_endpoint setting is missing the http:// or https:// protocol prefix. This is the permanent fix for Terraform Enterprise v202312-1 and later.

1. Set the S3 endpoint value, ensuring it includes the protocol prefix.

   $ replicatedctl app-config set s3_endpoint --value https://your-endpoint-url.com

2. Apply the new configuration to restart the application.

   $ replicatedctl app apply-config

Solution 3: Adjust Endpoint for "NoSuchKey" Error

This error typically occurs due to an incorrect endpoint configuration for your object storage provider.

• For AWS S3: If you are using native AWS S3, the S3 endpoint setting should be empty. Unset it with the following command.

  $ replicatedctl app-config set s3_endpoint --value ""

• For S3-Compatible Storage: If you are using an S3-compatible endpoint, ensure the bucket name is not included in the S3 endpoint URL itself.

After adjusting the configuration, apply the changes to restart the application.

$ replicatedctl app apply-config

Outcome

After applying the appropriate solution, Terraform Enterprise should start successfully.