Terraform Enterprise Fails to Start With Custom S3 Endpoint When Consolidated Services Are Enabled

Problem

When Terraform Enterprise is started with a custom S3 endpoint without a protocol prefix (bare hostname) in consolidated services mode with Replicated, the application fails to start.  Consolidated services is automatically enabled when upgrading from earlier Terraform Enterprise version to v202309-1.

A log entry similar to the following will be present in the terraform-enterprise container:

[ERROR] terraform-enterprise: startup: error="failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"/s3.your-object-storage-provider.com/tfe-bucket?delimiter=%2F&list-type=2\": unsupported protocol scheme \"\"

[ERROR] terraform-enterprise: startup: error=\"failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: , HostID: , api error NoSuchKey: The specified key does not exist.\"

Prerequisites

• Terraform Enterprise (Replicated deployment) with Consolidated Services Architecture
• Terraform Enterprise <=v202312-1 ("unsupported protocol scheme" error only) 
• Terraform Enterprise Flexible Deployment Options

Cause

Terraform Enterprise with Consolidated Services Architecture and Terraform Enterprise Flexible Deployment Options using AWS SDK v2, which introduces some changes to S3 endpoint resolution. Previously, a user could supply a S3 endpoint FQDN with no protocol scheme however, due to a bug with how the S3 endpoint is utilized in the Terraform Enterprise application when consolidated services is enabled, it no results in the "unsupported protocol scheme" error- this is fixed in v202312-1. The "NoSuchKey: The specified key does not exist." error ties down to resolution of custom endpoints when using virtual-hosted–style URIs versus path-style URIs.

Solutions

Option one:

If the Terraform Enterprise application is a Replicated deployment <=v202312-1, as a temporary workaround, disable consolidated services mode.

1. Disable consolidated services.
   

   replicatedctl app-config set consolidated_services_enabled --value 0

2. Restart Terraform Enterprise with the application with the updated configuration
   

   replicatedctl app apply-config

Option two:

Correct the configuration to account for the behavior using one of the procedures below, depending on the error.

"unsupported protocol scheme"

1. Include the protocol prefix in your Replicated s3_endpoint setting.
2. Manually set the endpoint including prefix with the following command
   

   replicatedctl app-config set s3_endpoint --value https://your-endpoint-url.com

3. Restart the application 

   replicatedctl app apply-config

"NoSuchKey: The specified key does not exist."

1. If using native S3, unset the S3 endpoint setting
   

   replicatedctl app-config set s3_endpoint --value ""

2. If using s3-compatible endpoint, remove the bucket name from the S3 endpoint
3. Restart the application 

   replicatedctl app apply-config

Outcome:

Once one of these solutions is followed, the application should start successfully. Otherwise, please reach out to HashiCorp Support.