Terraform Enterprise Mounted Disk PostgreSQL Database Upgrade from v14 to v16 – HashiCorp Help Center

Introduction

This article details the recovery process for a failed PostgreSQL database upgrade in Terraform Enterprise (TFE) version v202406-1. This upgrade process applies to TFE installations in mounted disk mode that are upgrading from PostgreSQL v14 to v16. The v202406-1 release also makes PostgreSQL v16 the default version for new TFE installations.

Key Directories

During the upgrade process, TFE uses several key directories on the host machine:

Primary Data Directory: /$DISKPATH/postgres/pgdata
Temporary Upgrade Directories: /$DISKPATH/postgres/14/data and /$DISKPATH/postgres/16/data
PostgreSQL v14 Backup: /$DISKPATH/postgres-backup/14

Problem

During the automated upgrade to PostgreSQL v16, the process fails. The Terraform Enterprise application will not start, and the PostgreSQL log at /$DISKPATH/log/postgres.log contains the following error message:

Error: Failed upgrading the PostgreSQL data located within the /var/lib/terraform-enterprise/postgres/pgdata directory on the host machine.

If you'd like to try the PostgreSQL upgrade again, please restore the PostgreSQL 14 files to the /var/lib/terraform-enterprise/postgres/pgdata directory on the host machine.

Unless the PostgreSQL backup was skipped, a backup of the PostgreSQL 14 files should be located within the /var/lib/terraform-enterprise/postgres-backup/14 directory on the host machine.

Once finished, remove the file /var/lib/terraform-enterprise/postgres/pgdata/.pg_upgrade_failed on the host machine to tell this program to upgrade the PostgreSQL data next time it runs.

Restart the Terraform Enterprise application to run this program again.

Cause

The upgrade process automatically creates a backup of the PostgreSQL v14 database, then attempts a test upgrade using temporary directories. If this test upgrade fails, the process halts and creates a .pg_upgrade_failed file in the /$DISKPATH/postgres/pgdata directory. This file prevents subsequent startup attempts from re-running the failed upgrade until you perform a manual cleanup.

Solution

To recover from the failed upgrade, you must restore the PostgreSQL v14 database from the backup and remove the failure indicator file. This allows Terraform Enterprise to re-attempt the upgrade on the next startup.

Procedure

Ensure the primary PostgreSQL data directory is empty. This prepares it for the restored data.
```
$ rm -rf /$DISKPATH/postgres/pgdata/*
```
Restore the PostgreSQL v14 database by copying the backup files into the now-empty data directory.
```
$ cp -r /$DISKPATH/postgres-backup/14/* /$DISKPATH/postgres/pgdata
```
Remove the .pg_upgrade_failed file. This signals to Terraform Enterprise that it is safe to attempt the upgrade again.
```
$ rm /$DISKPATH/postgres/pgdata/.pg_upgrade_failed
```

Outcome

After completing these steps, you can restart the Terraform Enterprise application. The startup process will detect the restored v14 data and re-initiate the upgrade to PostgreSQL v16.

Additional Information

For more details on Terraform Enterprise administration and upgrades, please refer to the official Terraform Enterprise documentation.