Removing a Stale Migration Lock Value in Redis – HashiCorp Help Center

Problem

If a Terraform Enterprise instance crashes or stops prematurely during a database migration, other instances in the deployment may become unresponsive or wait for up to one hour before they can proceed with their own migrations.

Cause

Terraform Enterprise uses a Redis lock to ensure that only one instance runs database migrations at a time, preventing data corruption in multi-instance deployments. In cases where an instance is forcefully terminated while holding this lock, the lock value in Redis may not be cleared. As a result, other Terraform Enterprise instances will wait for the lock to expire before they can run migrations.

Solution

The migration lock is created with a one-hour time-to-live (TTL). In the event of a crashed migration, the lock will automatically expire one hour after the process began. To resolve the issue sooner, you can manually remove the lock from Redis by following the procedure below.

Prerequisites

Shell access to the host machine running the Terraform Enterprise instance.
The name or ID of the Terraform Enterprise container (e.g., tfe-atlas).

Procedure

Connect to the Redis instance running within the Terraform Enterprise container. The command differs based on your Terraform Enterprise version.
For TFE version v202404-2 and newer:
```
$ docker exec -ti <TFE_CONTAINER> bash -c '. /var/run/terraform-enterprise/atlas/atlas-env && redli -u $REDIS_URL'
```
For TFE versions older than v202404-2:
```
$ docker exec -ti <TFE_CONTAINER> bash -c '. atlas-env && redli -u $REDIS_URL'
```
Once connected to the Redis command-line interface, delete the tfe_migration_lock key. You should see an integer response of 1, indicating one key was deleted.
```
> del tfe_migration_lock
(integer) 1
```

Outcome

Once you remove the key from Redis, any Terraform Enterprise nodes waiting to run migrations will be able to acquire the lock and proceed normally.

Additional Information

For more details on Terraform Enterprise administration, please refer to the official documentation.
For information on interacting with Redis, consult the Redis commands documentation.