Problem
When using the Restore API endpoint to restore a backup of a Terraform Enterprise (TFE) installation, the application fails to start and reports the following error in the logs.
[ERROR] tfe-task-worker: error running database migrations: err="migrate: no migration found for version X: read down for version X migrations: file does not exist"
Cause
This error occurs when the backup was created from a different version of Terraform Enterprise than the version you are restoring to. This commonly happens when attempting to roll back to a previous version of Terraform Enterprise after an upgrade.
Solution
To resolve this issue, you must manually rename the existing PostgreSQL database. This action forces Terraform Enterprise to create a new, clean database on the next startup, allowing the restore operation to complete successfully.
Prerequisites
- The
postgresql-clientpackage must be installed on the Terraform Enterprise host to runpg_dumpandpsqlcommands.-
Debian/Ubuntu:
$ sudo apt-get install postgresql-client
-
RHEL/CentOS:
$ sudo yum install postgresql-client
-
-
You will need your PostgreSQL connection details (
PG_USERNAME,PG_PASSWORD,PG_NETLOCATION,PG_DBNAME). You can retrieve these values by running the following command.$ replicatedctl app-config export --hidden
Procedure
-
Stop the Terraform Enterprise application.
$ replicatedctl app stop
-
Monitor the application status until it reports a
stoppedstate.$ watch replicatedctl app status
-
Create a local backup of the current database.
$ pg_dump postgres://<PG_USERNAME>:<PG_PASSWORD>@<PG_NETLOCATION>:5432/<PG_DBNAME>?sslmode=require > tfe_pgdump.backup
-
Connect to the PostgreSQL server using a default database like
postgres.$ psql postgres://<PG_USERNAME>:<PG_PASSWORD>@<PG_NETLOCATION>:5432/postgres?sslmode=require
-
Rename the current Terraform Enterprise database by appending
_oldto its name.ALTER DATABASE "CurrentDBName" RENAME TO "CurrentDBName_old";
If this command fails due to active connections, terminate them first and then retry the rename command.
SELECT pg_terminate_backend (pg_stat_activity.pid) \ FROM pg_stat_activity \ WHERE pg_stat_activity.datname = 'CurrentDBName';
-
Create a new, empty database with the original name.
CREATE DATABASE "OriginalDBName";
-
Exit the
psqlsession.exit
-
Start the Terraform Enterprise application.
$ replicatedctl app start
If the application fails to start, review the application logs for further errors before proceeding.
- Execute the API request to the restore endpoint again to restore the backup.
-
Restart the application to apply the restored data.
$ replicatedctl app stop $ watch replicatedctl app status ## Wait for the state to become 'stopped' $ replicatedctl app start
If the application fails to start after the restore, review the application logs for further errors.
- Verify that the data was restored successfully by logging into the Terraform Enterprise UI.
Optional Cleanup
After confirming a successful restore, you can remove the old database and the local backup file.
-
Connect to the PostgreSQL server again.
$ psql postgres://<PG_USERNAME>:<PG_PASSWORD>@<PG_NETLOCATION>:5432/postgres?sslmode=require
-
Drop the old database.
DROP DATABASE "DatabaseName_old";
-
Exit the
psqlsession.exit
-
Delete the local database backup file.
$ rm tfe_pgdump.backup
Outcome
After completing the procedure, Terraform Enterprise starts successfully with all data from the backup restored.