Terraform Enterprise fails to start: Container tfe-postgres-upgrade failed – HashiCorp Help Center

Problem

After upgrading a Terraform Enterprise instance in mounted disk mode to release v202207-1 or v202207-2, the application fails to start. The tfe-postgres-upgrade container fails during the startup sequence.

Prerequisites

Terraform Enterprise configured in mounted disk operational mode.
An upgrade to release v202207-1 or v202207-2 has been performed.

Cause

The upgrade process from PostgreSQL 12 to PostgreSQL 14 fails because the startup script cannot find the pg_hba.conf file in the expected backup location. This results in a cannot stat '/backup/12/pg_hba.conf': No such file or directory error in the container logs, which prevents the database from starting correctly.

You may see log entries similar to the following from the tfe-postgres-upgrade container.

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
LOG:  starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
LOG:  listening on IPv4 address "0.0.0.0", port 5432
LOG:  listening on IPv6 address "::", port 5432
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG:  database system was shut down at 2023-03-31 14:08:10 UTC
LOG:  database system is ready to accept connections
LOG:  received smart shutdown request
LOG:  background worker "logical replication launcher" (PID 258) exited with exit code 1
LOG:  shutting down
LOG:  database system is shut down
cp: cannot stat '/backup/12/pg_hba.conf': No such file or directory
rm: cannot remove '/var/lib/postgresql/data': Device or resource busy

Solution

To resolve this issue, you must manually edit the PostgreSQL pg_hba.conf file to add a required authentication rule and then restart the Terraform Enterprise application.

Procedure

Connect to the Terraform Enterprise host via SSH.

Retrieve the path to the mounted disk data directory.

## The output of this command is the path you will use in the next step.
$ replicatedctl app-config export --template '{{.disk_path.Value}}'
/opt/hashicorp/data

Open the pg_hba.conf file for editing, using the path from the previous step. For example, if your path is /opt/hashicorp/data, the command is as follows.
```
$ sudo vim /opt/hashicorp/data/postgres/pgdata/pg_hba.conf
```
Add the following line to the end of the pg_hba.conf file, then save your changes and exit the editor.
```
host all all all md5
```
Stop the Terraform Enterprise application.
```
$ replicatedctl app stop -f
```
Start the Terraform Enterprise application to allow the upgrade to complete.
```
$ replicatedctl app start
```

Outcome

During the application startup, monitor the logs of the tfe-postgres-setup container. The process is successful when you see the following message.

PostgreSQL data upgrade completed successfully! Exiting.

Additional Information