Problem
When Terraform Enterprise is started with a custom S3 endpoint without a protocol prefix (bare hostname) in consolidated services mode with Replicated, the application fails to start. Consolidated services is automatically enabled when upgrading from earlier Terraform Enterprise version to v202309-1.
A log entry similar to the following will be present in the terraform-enterprise container:
[ERROR] terraform-enterprise: startup: error="failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"/s3.your-object-storage-provider.com/tfe-bucket?delimiter=%2F&list-type=2\": unsupported protocol scheme \"\"
[ERROR] terraform-enterprise: startup: error=\"failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: , HostID: , api error NoSuchKey: The specified key does not exist.\"
Prerequisites
- Terraform Enterprise (Replicated deployment) with Consolidated Services Architecture
- Terraform Enterprise <=v202312-1 ("unsupported protocol scheme" error only)
- Terraform Enterprise Flexible Deployment Options
Cause
Terraform Enterprise with Consolidated Services Architecture and Terraform Enterprise Flexible Deployment Options using AWS SDK v2, which introduces some changes to S3 endpoint resolution. Previously, a user could supply a S3 endpoint FQDN with no protocol scheme however, due to a bug with how the S3 endpoint is utilized in the Terraform Enterprise application when consolidated services is enabled, it no results in the "unsupported protocol scheme" error- this is fixed in v202312-1. The "NoSuchKey: The specified key does not exist." error ties down to resolution of custom endpoints when using virtual-hosted–style URIs versus path-style URIs.
Solutions
Option one:
If the Terraform Enterprise application is a Replicated deployment <=v202312-1, as a temporary workaround, disable consolidated services mode.
- Disable consolidated services.
replicatedctl app-config set consolidated_services_enabled --value 0
- Restart Terraform Enterprise with the application with the updated configuration
replicatedctl app apply-config
Option two:
Correct the configuration to account for the behavior using one of the procedures below, depending on the error.
"unsupported protocol scheme"
- Include the protocol prefix in your Replicated
s3_endpoint
setting. - Manually set the endpoint including prefix with the following command
replicatedctl app-config set s3_endpoint --value https://your-endpoint-url.com
- Restart the application
replicatedctl app apply-config
"NoSuchKey: The specified key does not exist."
- If using native S3, unset the S3 endpoint setting
replicatedctl app-config set s3_endpoint --value ""
- If using s3-compatible endpoint, remove the bucket name from the S3 endpoint
- Restart the application
replicatedctl app apply-config
Outcome:
Once one of these solutions is followed, the application should start successfully. Otherwise, please reach out to HashiCorp Support.