After upgrade to 202402-1 Terraform Enterprise runs are stuck in Pending and sidekiq logs show the error FATAL: password authentication failed for user

Introduction

Problem

All the workspaces in TFE are not able to plan or apply, because the runs are stuck in a Pending state.

When checking the TFE logs, the following error message is present in the sidekiq container:

sidekiq.log:2024-03-14T13:18:13.215Z pid=13658 tid=18vi WARN: ActiveRecord::
ConnectionNotEstablished: connection to server at "10.114.11.31", port 5432 
failed: FATAL: password authentication failed for user "postgres" 
sidekiq.log:connection to server at "10.114.11.31", port 5432 failed: FATAL: 
password authentication failed for user "postgres"

Prerequisites

• TFE 202402-1

Cause

• The DB Password contains special characters and is causing the sidekiq component of TFE to crash
• This is a known bug in Terraform Enterprise 202402-1 and is being tracked internally with our Dev Engineering

Workaround:

• TFE FDO
  • Change the DB password to a password without any special characters
  • Redeploy TFE

• TFE Replicated 
  • Change the DB password to a password without any special characters
  • Restart TFE

Solutions:

• Upgrade to the TFE v202402-2 

Outcome

For now, the workaround of having a DB password without any special characters will resolve the issue and runs will start working again.

Additional Information

• Terraform Enterprise Releases