Terraform Enterprise Fails to Start With Custom S3 Endpoint When Consolidated Services Are Enabled

Problem

When Terraform Enterprise is started with a custom S3 endpoint without a protocol prefix (bare hostname) in consolidated services mode with Replicated, the application fails to start.  Consolidated services is automatically enabled when upgrading from earlier Terraform Enterprise version to v202309-1.

A log entry similar to the following will be present in the terraform-enterprise container:

[ERROR] terraform-enterprise: startup: error="failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"/s3.your-object-storage-provider.com/tfe-bucket?delimiter=%2F&list-type=2\": unsupported protocol scheme \"\""

Cause

• This is due to a bug with how the S3 endpoint is utilized in the Terraform Enterprise application when consolidated services is enabled.

Solutions: (Select one)

1. Stop the application

   replicatedctl app stop

2. Check the status of the application

   replicatedctl app status

3. Once application is stopped, disable consolidated services.

   replicatedctl app-config set consolidated_services_enabled --value 0

4. Include the protocol prefix in your Replicated s3_endpoint setting.

5. Manually set the endpoint including prefix with the following command
   replicatedctl app-config set s3_endpoint --value https://your-endpoint-url.com

6. Restart the application 

   replicatedctl app apply-config

Outcome:

Once one of these solutions is followed, the application should start successfully. Otherwise, please reach out to HashiCorp Support.