Introduction
Problem
Terraform Enteprise will not start correctly and we see the following error in the Vault log of Terraform Enterprise
Initializing Vault.
{"@level":"debug","@message":"lock id","@module":"lock-cmd","@timestamp":"2025-02-06T06:17:46.961121Z","id":123456789}
{"@level":"debug","@message":"acquiring lock","@module":"lock-cmd","@timestamp":"2025-02-06T06:17:46.962978Z","timeout":600000000000}
{"@level":"debug","@message":"attempt to acquire lock","@module":"lock-cmd.poll-locker","@timestamp":"2025-02-06T06:17:46.964310Z","res":false}
These messages can be found in the container at /var/log/terraform-enterprise/vault.log
Prerequisites
- Terraform Enterprise and using an external PostgreSQL database
Cause
When the Terraform Enterprise container starts, its internal services initialize sequentially. One of these services is Vault. As Vault starts, it connects to the PostgreSQL database backend to create a lock record, ensuring that no other Vault instance is starting simultaneously.
If a lock already exists in the PostgreSQL database, Vault will wait for it to be released before proceeding. However, if the lock remains due to a previous crash or other issues, manual removal from PostgreSQL may be required.
Solution:
- Connect to your PostgreSQL database
# Example from the Terraform Enterprise container itself
psql postgres://$TFE_DATABASE_USER:$TFE_DATABASE_PASSWORD@$TFE_DATABASE_HOST/$TFE_DATABASE_NAME?$TFE_DATABASE_PARAMETERS
- Check if there are any logs that are matching that of the id that is in the Vault log
{"@level":"debug","@message":"lock id",
"@module":"lock-cmd","@timestamp":"2025-02-06T06:17:46.961121Z",
"id":123456789}
Use the following command to see the locks
select * from pg_locks;
- Verify from the output that this lock is not coming from another Terraform Enterprise environment that is still in the startup process.
- Unlock the id from the Vault log
SELECT pg_advisory_unlock(123456789);
- Vault should immediately continue starting and Terraform Enterprise should become available
Outcome
After the lock is removed from Terraform Enterprise the Vault service should correctly start.