TFE: Failed to enqueue cost estimate or Sentinel Policy check error after upgrade of a mounted disk instance

Introduction

Specific Terraform Enterprise configurations can lead to a bug that fails to create a required database schema, preventing aspects of the Terraform Enterprise application from working correctly. 

Problem

When using a mounted disk mode instance that was upgraded from a previous version up to Terraform Enterprise v202212-x, a bug prevents the creation of a required database schema. Once this issue manifests you can run into a number of errors, one of which is cost estimation throwing the error Failed to enqueue cost estimate or Sentinel Policy Checks failing. 

Prerequisites

• Existing Mounted disk instance of Terraform Enterprise
  • Instance upgraded to v202212-x

Cause

If you were to generate a support bundle within the tfe-task-worker.stdout file you would likely see a similar error like the below example. 

"error","@message":"error running database migrations", 3"@module":
"tfe-task-worker","@timestamp":"2022-12-15T17:02:37.498823Z","err":"database 
driver: sql: Scan error on column index 0, name \"current_schema\": converting 
NULL to string is unsupported in 5line 0: SELECT CURRENT_SCHEMA()"}

Solution:

• SSH into your Terraform Enterprise instance
• Connect to PostgreSQL using docker exec -it tfe-postgres psql -U hashicorp.
• List the schemas using \dn.
  • The expected output is below. If your support bundle contains errors like the above example, we're assuming that within your output, the task_worker schema will be missing.

hashicorp=# \dn
     List of schemas
    Name     |   Owner
-------------+-----------
 rails       | hashicorp
 registry    | hashicorp
 task_worker | hashicorp
 vault       | hashicorp
(4 rows)

• Create the missing task_worker schema with CREATE SCHEMA IF NOT EXISTS task_worker AUTHORIZATION hashicorp;
• List the schemas again using \dn. Observe that task_worker is now listed.
• Exit using quit or exit and restart the TFE application.

Outcome

Once the task_worker schema is created, and the Terraform Enterprise application is rebooted, functionality should be restored throughout the application. 

Additional Information

• If you continue to experience issues, please contact HashiCorp Support.