Problem
Terraform Enterprise fails to start with the following log trail, indicating that the terraform-enterpise process has exited early, causing Fluent Bit to fail with "[error] could not open configuration file, aborting.".
2024-05-10 15:01:43,677 INFO stopped: terraform-enterprise (terminated by SIGTERM)
2024-05-10 15:01:44,680 INFO success: fluent-bit entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[2024/05/10 15:01:45] [error] could not open configuration file, aborting.
2024-05-10 15:01:45,523 INFO stopped: fluent-bit (exit status 1)
A TLS verification error is logged to the /var/log/terraform-enterprise/terraform-enterprise.log file in the container, referencing the configured external object storage endpoint's certificate.
root@ip-10-0-30-207:/opt/terraform-enterprise# docker compose cp tfe:/var/log/terraform-enterprise/terraform-enterprise.log .
[+] Copying 1/0
✔ terraform-enterprise-tfe-1 copy terraform-enterprise-tfe-1:/var/log/terraform-enterprise/terraform-enterprise.log to . Copied 0.0s
root@ip-10-0-30-207:/opt/terraform-enterprise# cat terraform-enterprise.log
panic: error loading config: failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://tfe-minio:9000/tfe?delimiter=%2F&list-type=2": tls: failed to verify certificate: x509: certificate signed by unknown authority
goroutine 1 [running]:
main.main()
/home/runner/work/terraform-enterprise/terraform-enterprise/tools/service-setup/main.go:37 +0x396
Prerequisites
- Terraform Enterprise v202404-2
- External S3-compatible object storage using a self signed or non-publicly trusted CA certificate which has been added to the CA bundle
Cause
This issue is caused by a bug introduced in Terraform Enterprise v202404-2. When Terraform Enterprise starts, the terraform-enterprise process sets up its trusted certificates using the contents of the CA bundle setting. In v202404-2, changes were made to the startup order, causing a request to the configured s3-compatible object storage endpoint before this certificate setup. As a result, Terraform Enterprise cannot verify the endpoint's certificate, resulting in a failure.
Solution
This bug is fixed in v202405-1. Upgrading to this release or later will resolve the issue. If an upgrade is not preferred, there are two alternatives, depending on the deployment option used.
- Pin the previous release by modifying the image tag (Flexible Deployment Options)
- Repave the server using the previous release (Replicated)
This will enable successful startup on the previous release, as the application will not have progressed far enough into startup to run the database migrations for v202404-2.
Additional Information
If you continue to experience issues, please contact HashiCorp Support.