This article details the Terraform Enterprise PostgreSQL database upgrade in the v202406-1 TFE Release. This upgrade targets TFE installs in mounted disk mode that are running v14 of PostgreSQL. This release will also make PostgreSQL v16 the default version of PostgreSQL for brand new TFE installs.
Folders to Note on the Host
Where PostgreSQL lives in the TFE container:
/$DISKPATH/postgres/pgdata
Temporary directories for testing the database upgrade:
/$DISKPATH/postgres/14/data/$DISKPATH/postgres/16/data
Backup of PostgreSQL 14:
/$DISKPATH/postgres-backup/14
Process Overview
- If there's no evidence of previous upgrade failures, the program will ensure enough space is available to create a PostgreSQL v14 backup, and then creates the backup.
- The program creates v14 and v16 upgrade directories to test out the upgrade itself. The program copies the real v14 database data into a temporary directory and attempts the v16 upgrade (targeting the temporary v16 data directory).
- If the upgrade is successful, the program will move the temporary v16 data post-upgrade to the actual data directory on the host.
- If the upgrade fails, the program will place a
.pg_upgrade_failedfile in the data directory to indicate that some cleanup is needed before attempting another upgrade.
Upgrade Failure Recovery
The following commands outline the process to restore the v14 PostgreSQL database should the upgrade to v16 fail. A customer will experience an upgrade failure and logs (at /$DISKPATH/log/postgres.log) will contain this blurb:
Error: Failed upgrading the PostgreSQL data located within the /var/lib/terraform-enterprise/postgres/pgdata directory on the host machine.
If you'd like to try the PostgreSQL upgrade again, please restore the PostgreSQL 14 files to the /var/lib/terraform-enterprise/postgres/pgdata directory on the host machine.
Unless the PostgreSQL backup was skipped, a backup of the PostgreSQL 14 files should be located within the /var/lib/terraform-enterprise/postgres-backup/14 directory on the host machine.
Once finished, remove the file /var/lib/terraform-enterprise/postgres/pgdata/.pg_upgrade_failed on the host machine to tell this program to upgrade the PostgreSQL data next time it runs.
Restart the Terraform Enterprise application to run this program again.
If there are still issues after following the above instructions, please contact HashiCorp Support.
1. Make sure the data directory is empty:
rm -rf /$DISKPATH/postgres/pgdata/*
2. Restore the PostgreSQL v14 database by copying the PostgreSQL v14 backup files to the data directory:
cp -r /$DISKPATH/postgres-backup/14/* /$DISKPATH/postgres/pgdata
3. Delete the .pg_upgrade_failed file from the host to attempt the PostgreSQL upgrade again:
rm /$DISKPATH/postgres/pgdata/.pg_upgrade_failed