Resolving TLS Handshake failure to S3 Object Storage Endpoint on TFE v202406-1 – HashiCorp Help Center

Issue Overview

When upgrading Terraform Enterprise (TFE) to v202406-1, you may encounter the following TLS handshake failure during startup when Terraform Enterprise attempts to list objects in S3 object storage.

[root@tfe ~]# podman exec -it terraform-enterprise-terraform-enterprise /bin/bash

terraform-enterprise@terraform-enterprise:/$ tail -n 100 -f /var/log/terraform-enterprise/terraform-enterprise.log

sudo: unable to send audit message: Operation not permitted

panic: error loading config: failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get https://s3.yourcompany.com/qatfe?delimiter=%2F&list-type=2 : remote error: tls: handshake failure

goroutine 1 [running]:

main.main()

/home/runner/work/terraform-enterprise/terraform-enterprise/tools/service-setup/main.go:37 +0x34e

2024-10-28T20:39:38.971Z [ERROR] terraform-enterprise: startup: error="error setting up TFE services: error running service-setup: exit status 2"

Cause

This error is caused by changes in the default TLS cipher suites introduced in Go 1.22, which Terraform Enterprise v202406-1 uses. The handshake failure occurs because the hosted S3-compatible storage is using an incompatible cipher suite with the TFE. The updated Go version removes support for RSA key exchange ciphers, such as DHE-RSA-AES256-SHA, favoring ECDHE-based ciphers for enhanced security and performance.

Suggested Resolution

Option A

Reconfigure the cipher suite on the hosted S3-compatible storage to ECDHE-based ciphers such as one of the following.

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_CHACHA20_POLY1305_SHA256

Option B

If reconfiguring is not possible, a temporary workaround would be to add the GODEBUG environment variable to allow RSA key exchange ciphers and reattempt the upgrade.

Steps to Fix

Verify the S3 Server's Cipher Suite: Run the following command to identify the TLS cipher suite used by the server:
```
openssl s_client -connect s3.yourcompany.com:443
```
Look for the Cipher value in the output. For example:
```
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES256-SHA
```
This example shows an incompatible cipher being used
Re-enable RSA Key Exchange Ciphers: Update the TFE configuration to include the GODEBUG environment variable:
```
GODEBUG="tlsrsakex=1"
```
Save the changes and redeploy Terraform Enterprise.
Reattempt the upgrade to v202406-1: The container should not crash and the upgrade should proceed normally.

References

Need Further Assistance?

If you encounter further issues or need additional help, please contact support with your logs and configuration details.