Terraform Enterprise startup fails as Vault attempts to acquire a lock

Problem

Terraform Enterprise may fail to start, and its internal Vault service log displays errors related to acquiring a database lock. The service appears to hang while waiting for the lock to be released.

You can find these messages in the Vault log file, located at /var/log/terraform-enterprise/vault.log inside the container. The log entries resemble the following output, indicating that Vault is continuously attempting to acquire a lock.

{
  "@level": "debug",
  "@message": "lock id",
  "@module": "lock-cmd",
  "@timestamp": "YYYY-MM-DDTHH:MM:SSZ",
  "id": 123456789
}
{
  "@level": "debug",
  "@message": "acquiring lock",
  "@module": "lock-cmd",
  "@timestamp": "YYYY-MM-DDTHH:MM:SSZ",
  "timeout": 600000000000
}
{
  "@level": "debug",
  "@message": "attempt to acquire lock",
  "@module": "lock-cmd.poll-locker",
  "@timestamp": "YYYY-MM-DDTHH:MM:SSZ",
  "res": false
}

Prerequisites

• Terraform Enterprise configured with an external PostgreSQL database.

Cause

When a Terraform Enterprise container starts, its internal services initialize sequentially. The internal Vault service connects to the PostgreSQL database backend and creates an advisory lock to ensure that no other Vault instance attempts to initialize at the same time.

If a previous startup process crashed or was terminated improperly, this lock may not have been released. The new Vault instance will detect the existing lock and wait indefinitely for it to be cleared, preventing Terraform Enterprise from starting.

Solution

Follow these steps to manually remove the advisory lock from the PostgreSQL database.

1. Connect to your PostgreSQL database. You can use the following command from within the Terraform Enterprise container, which uses predefined environment variables.

   # psql postgres://$TFE_DATABASE_USER:$TFE_DATABASE_PASSWORD@$TFE_DATABASE_HOST/$TFE_DATABASE_NAME?$TFE_DATABASE_PARAMETERS

2. Identify the lock ID from the Vault log. In the example log output, the ID is 123456789.

3. Verify that the lock is not from another active Terraform Enterprise instance that is still starting up. You can query for active locks.

   select * from pg_locks;

4. Remove the advisory lock using the ID from the Vault log. Replace 123456789 with the actual ID from your log file.

   SELECT pg_advisory_unlock(123456789);

Outcome

After you remove the lock, the Vault service continues its startup sequence, and Terraform Enterprise should become available.