Problem
When upgrading to or installing Terraform Enterprise v202307-1, the application may fail to start. The tfe-atlas container repeatedly logs messages indicating it is waiting for a database migration.
waiting on database to reach 20230628000004
Additionally, the tfe-migrations container logs show a Dangerous operation detected error related to adding an index concurrently in PostgreSQL.
execing command; /usr/local/bundle/bin/bundle [exec ruby /app/atlas-migration-wrapper.rb]
rake aborted!
StandardError: An error has occurred, all later migrations canceled:=== Dangerous operation detected #strong_migrations ===
Adding an index concurrently can cause silent data corruption in Postgres 14.0 to 14.3.
Upgrade Postgres before adding new indexes, or wrap this step in a safety_assured { ... } block
to accept the risk.
/app/db/migrate/20230628000001_add_index_for_fks.rb:9:in `change'
/app/config/initializers/active_record_migrations.rb:15:in `exec_migration'
/usr/local/bundle/gems/bundler-2.3.25/lib/bundler/cli/exec.rb:58:in `load'
## ...stack trace truncated for brevity...Cause
The error occurs because of a known bug in PostgreSQL versions 14.0 through 14.3 that can cause silent data corruption when multiple indices are added concurrently.
The Terraform Enterprise migration file, 20230628000001_add_index_for_fks.rb, performs this type of operation. The migrations library correctly identifies this action as dangerous on the affected PostgreSQL versions and halts the process to prevent potential data corruption.
This issue was fixed in PostgreSQL version 14.4.
Solution
To resolve this issue, you must upgrade the external PostgreSQL database instance used by Terraform Enterprise to a version that is not affected by this bug.
- Upgrade PostgreSQL to version
14.4or a later14.xrelease. - Alternatively, upgrade to a major version, such as PostgreSQL
15.xor newer.
After upgrading the database, restart the Terraform Enterprise application to allow the migrations to complete successfully.