Error in Database Permissions Post TFE Upgrade Causes UI Failure

Introduction

This article outlines the troubleshooting steps and resolution for an issue where the Terraform Enterprise (TFE) UI failed to load after upgrading from version v202406-1 to v202501-1. It details the root cause, diagnostics performed, and corrective actions taken to restore service availability.

Problem

After upgrading TFE from v202406-1 to v202501-1, the UI did not load, and multiple errors were observed in logs and containers, affecting application availability.
 

Cause

This issue was primarily caused due to:

1. Insufficient database privileges for the database user, particularly on the custom type service_account_type

2. Improper proxy configuration, which led to Vault startup failures

Errors encountered:

• Application startup error:

  [ERROR] terraform-enterprise.tfectl-command-queue: failed to clean queue: error="entity not found"

• Sidekiq log and atlas.log:

  PG::InsufficientPrivilege: ERROR: must be owner of type service_account_type

Solutions

Solution 1: Validate and Update Database Permissions

1. Coordinate with the DB team to confirm the user assigned to TFE has ownership permissions for all custom types and schemas.

2. If not, run the following command to assign ownership:

   ALTER TYPE service_account_type OWNER TO <postgres_username_used_by_TFE>;

Solution 2: Clean Up and Restart TFE Services

1. Delete and recreate the replicated container.

2. Ensure the TFE_DISK_PATH environment variable is unset when using external storage.

3. Use Docker Compose to bring up the services:

   docker compose down
   docker compose up -d

Outcome

After granting ownership privileges on service_account_type to the TFE DB user and correcting proxy configurations, the application restarted successfully and the UI became operational.

To confirm resolution:

• UI loads without errors

• No permission-related logs appear in atlas.log or sidekiq.log

• Vault is unsealed and accessible

• No Sidekiq queue errors appear

If the issue persists, review database schema access, Docker container logs, and Vault configuration again.

Additional Information

• TFE Documentation: Configuring TFE Storage and Database Access

• Database Role Management: Ensure the DB role used by TFE has sufficient privileges to manage all custom types and schemas introduced by migrations.