After upgrade to 202402-1 Terraform Enterprise runs are stuck in Pending and sidekiq logs show the error FATAL: password authentication failed for user

Problem

After upgrading to Terraform Enterprise v202402-1, runs are stuck in a Pending state and do not proceed. An inspection of the sidekiq container logs reveals a fatal password authentication error.

ActiveRecord::ConnectionNotEstablished: connection to server at "10.114.11.31", port 5432 failed: FATAL: password authentication failed for user "postgres"
connection to server at "10.114.11.31", port 5432 failed: FATAL: password authentication failed for user "postgres"

Prerequisites

• Terraform Enterprise v202402-1

Cause

This issue is caused by a known bug in Terraform Enterprise v202402-1 where the sidekiq component fails to correctly parse a database password that contains special characters.

Solutions

There are two options to resolve this issue. The recommended permanent solution is to upgrade Terraform Enterprise. If an immediate upgrade is not possible, you can use the workaround to change the database password.

Solution 1: Change the database password (Workaround)

Update the external database password to a new password that does not contain any special characters.

• For Flexible Deployment Options (FDO):
  1. Change the database password.
  2. Redeploy Terraform Enterprise with the updated password.
• For Replicated deployments:
  1. Change the database password.
  2. Restart Terraform Enterprise to apply the change.

Solution 2: Upgrade Terraform Enterprise (Recommended)

Upgrade to a version where this bug is resolved, such as TFE v202402-2 or a later version.

Outcome

After applying either the workaround or the upgrade, Terraform Enterprise will be able to connect to the database successfully, and runs will no longer be stuck in the Pending state.

Additional Information

• For a list of all versions, refer to the Terraform Enterprise Releases documentation.