Introduction
This article addresses an issue in Terraform Enterprise (TFE) where the application fails to start when using mixed authentication modes for the primary and Explorer databases. Specifically, the issue occurs when the main TFE database uses Google IAM passwordless authentication, while the Explorer database is configured with password-based authentication.
This configuration causes TFE pods to restart continuously, making the application unavailable.
Problem
When you attempt the following setup:
-
Main TFE database: IAM authentication enabled (
TFE_DATABASE_PASSWORDLESS_GOOGLE_USE_DEFAULT_CREDENTIALS="true") -
Explorer database: Password authentication enabled (
TFE_EXPLORER_DATABASE_PASSWORDis provided)
TFE is unable to start. Both pods enter a CrashLoopBackOff state, with 0/2 Ready status and failing health checks.
The logs contain the following key error messages, indicating that TFE attempts to use IAM authentication for both databases, which causes the Explorer database connection to fail because it expects a password.
FATAL: empty password returned by client (SQLSTATE 28P01)
PG::ConnectionBad: password authentication failed for user "admin-tfe-explorer"
Prerequisites
- You are running Terraform Enterprise version
v202402-1or similar with the Explorer feature enabled. - You are attempting to use passwordless IAM authentication for the primary TFE database while configuring the Explorer database with password-based authentication.
Cause
This behavior occurs because the TFE_DATABASE_PASSWORDLESS_GOOGLE_USE_DEFAULT_CREDENTIALS setting is global. When enabled, it forces all database connections within TFE, including the Explorer database, to use IAM passwordless authentication.
This is a known product limitation, as TFE does not currently support mixed database authentication modes (IAM for the primary database and password for the Explorer database). There is no environment variable available to configure IAM authentication for only one database independently.
HashiCorp Engineering has confirmed this limitation, and a bug has been filed to track the issue. This article will be updated when a fix is available in a future release.
Solutions
Until mixed-mode authentication is supported, you must use one of the following configurations.
Solution 1: Use Password Authentication for Both Databases
This is the recommended approach. Configure TFE to use password-based authentication for both the primary and Explorer databases by setting the following variables and leaving IAM authentication disabled.
TFE_DATABASE_PASSWORDTFE_EXPLORER_DATABASE_PASSWORD
Solution 2: Use IAM Authentication for the Main Database and Disable Explorer
If you must use IAM authentication, you can enable it for the main database but you must disable the Explorer database feature.
Enable IAM authentication:
TFE_DATABASE_PASSWORDLESS_GOOGLE_USE_DEFAULT_CREDENTIALS="true"
Ensure that no configuration variables for the Explorer database are set.