Problem
When upgrading to Terraform Enterprise v202103-1 or newer, the application may fail to start, and the Replicated console may display the following error:
Timeout waiting for event PostgreSQL Upgraded
This issue affects Proof of Concept and Mounted Disk installations that use the internally-managed PostgreSQL database.
Cause
Terraform Enterprise v202103-1 introduced an automated upgrade of its internal PostgreSQL database from version 9.5 to 12. Replicated, the service scheduler for Terraform Enterprise, has a predefined timeout for startup and upgrade events. If the database migration takes longer than this timeout period, Replicated marks the application as failed, even if the migration is still in progress.
The time required for the migration depends on variables such as database size and disk I/O performance.
Solution
Even if the startup process timed out, the underlying PostgreSQL upgrade container may still be running. Follow these steps to check its status and complete the upgrade.
1. Check Container Logs
Check the logs of the PostgreSQL upgrade container to determine its current status. Use the appropriate command for your installation type.
For Proof of Concept installations, run the following command:
$ sudo docker logs -f ptfe_postgres_upgrade
For Mounted Disk installations, run the following command:
$ sudo docker logs -f ptfe_postgres_upgrade_disk
2. Analyze Log Output
Review the log output to determine the next steps.
- If the logs are actively updating, the PostgreSQL data upgrade is still in progress. Continue to monitor the logs until the process completes or an error occurs.
-
If the last line of the logs shows the success message, the upgrade has completed successfully.
PostgreSQL data upgrade completed successfully! Exiting.
- If the logs show an error, the message should provide details to help diagnose the problem.
3. Restart Terraform Enterprise
If the log analysis confirms a successful upgrade, restart the Terraform Enterprise application to begin using the new PostgreSQL version.
$ replicatedctl app stop $ replicatedctl app start