Problem
In some Terraform Enterprise (TFE) environments, you may observe high memory usage in the external Redis instance. This can lead to performance degradation and cause slowness during Terraform runs.
Cause
Terraform Enterprise uses Redis to store data in memory as key-value pairs for fast access. TFE includes background jobs that periodically delete stale keys to manage memory utilization.
However, Redis also has its own memory management mechanisms. When a Redis instance in an Active-Active TFE deployment reaches 80% of its memory limit, it begins an eviction process to free up space. The default eviction policy for Active-Active databases is noeviction. If the Redis instance is undersized for the workload, memory usage can remain high despite TFE's cleanup jobs, as this policy prevents keys from being removed to make space for new data.
If memory usage continues to rise while keys are being evicted, the rate of eviction will increase to prevent an Out-of-Memory (OOM) state.
Solution
For Terraform Enterprise instances in an Active-Active configuration experiencing high Redis memory usage, the recommended solution is to increase the size of the Redis instance. A typical upgrade is from 8 GB to 16 GB, though the optimal size may vary based on your workload.
For this configuration, the recommended eviction policy remains noeviction to ensure data consistency. Increasing the available memory provides sufficient headroom for both TFE operations and Redis's internal management processes.
Additional Information
For more details on how Redis manages memory, refer to the official documentation.