Terraform Enterprise Flexible Deployment Options deployed on Google Cloud Platfrom fails Health Check with "error reaching http://127.0.0.1:7675/healthz"

Problem

After deploying Terraform Enterprise Flexible Deployment Options on Google Cloud Platform (GCP), the application UI may be accessible, but you encounter two issues:

1. The health check fails with a connection refused error.

   ERROR: error reaching http://127.0.0.1:7675/healthz: Get "http://127.0.0.1:7675/healthz": dial tcp 127.0.0.1:7675: connect: connection refused

2. A test Terraform run fails with an internal server error.

   There was an error connecting to Terraform Cloud. Please do not exit Terraform to prevent data loss! Trying to restore the connection...
   │ Error: Failed to create configuration version: internal server error
   │ Terraform Cloud returned an unexpected error.
   │ Sometimes this is caused by network connection problems,
   │ in which case you could retry the
   │ command.
   │ If the issue persists please open a support ticket to
   │ get help resolving the problem.

Cause

The issue occurs when the Google Cloud Storage JSON credentials are not provided in the Docker Compose configuration via the TFE_OBJECT_STORAGE_GOOGLE_CREDENTIALS environment variable.

You can confirm this by checking the archivist.log file for an error related to missing GCE metadata. You can access this log using one of the following methods:

• Execute a docker exec command to view the log directly from the container.

  $ docker exec -it terraform-enterprise-tfe-1 bash -c "cat /var/log/terraform-enterprise/archivist.log"

• Inspect the archivist.log file within a generated Support Bundle.

The log output will contain an error message similar to the following.

{
  "@level": "error",
  "@message": "failed to start server",
  "@module": "archivist",
  "@timestamp": "...",
  "err": "failed querying bucket attrs: Get \"https://XYZ?alt=json\u0026prettyPrint=false\u0026projection=full\": metadata: GCE metadata \"instance/service-accounts/default/token\" not defined"
}

Solutions

Solution 1: Provide Google Cloud Storage JSON Credentials

To resolve this issue, you must provide the necessary service account credentials to the Terraform Enterprise configuration and redeploy.

1. Obtain the required service account key in JSON format. The method depends on how you provisioned your GCP infrastructure.
   • Manual Configuration: Follow the Google Cloud documentation for creating service account keys.
   • Terraform Configuration: Use the private_key attribute from the google_service_account_key resource, as described in the Terraform code documentation.
2. Update your Docker Compose file to include the TFE_OBJECT_STORAGE_GOOGLE_CREDENTIALS environment variable, setting its value to the JSON key data you obtained.
3. Redeploy Terraform Enterprise with the updated Docker Compose file.

Outcome

After redeploying with the correct credentials, the health check command succeeds, and Terraform runs process correctly.

$ docker compose exec tfe tfe-health-check-status