Introduction
This article explains the expected Redis behavior when used with Sidekiq in a Terraform Enterprise (TFE) environment. It clarifies key management expectations and provides the recommended Redis configuration to prevent unintended key evictions.
Problem
When the Redis instance used by Terraform Enterprise reaches its maxmemory threshold, it begins evicting keys. While this behavior does not cause an application crash, it can lead to inconsistent job states for Sidekiq, which does not expect key eviction.
Cause
The issue stems from a mismatch between the default Redis configuration and the operational requirements of Sidekiq.
-
Redis Eviction Policy: When Redis reaches its configured
maxmemorylimit, it evicts keys based on its eviction policy (e.g.,volatile-lru,allkeys-lru). This is standard Redis behavior designed to manage memory. - Sidekiq's Requirements: Sidekiq is designed to manage its own keys within Redis and expects no external eviction behavior from the Redis instance. Eviction can undermine Sidekiq’s internal key lifecycle management, resulting in missing or inconsistent job data.
- Terraform Enterprise Key Management: Terraform Enterprise and its underlying Rails application delegate Redis key management primarily to Sidekiq. Therefore, the Redis configuration should align with Sidekiq's requirements.
Solutions
To ensure stability and predictable behavior, configure the Redis instance to align with Sidekiq’s operational model.
Solution 1: Set Redis Eviction Policy to 'noeviction'
The primary solution is to disable Redis's automatic key eviction, giving Sidekiq full control over key management. Update your Redis configuration to set the memory policy to noeviction.
Example Redis configuration:
maxmemory 2gb maxmemory-policy noeviction
Solution 2: Implement Memory Monitoring
Setting the policy to noeviction means that Redis will stop accepting write commands when it runs out of memory. To prevent this from causing application errors, you must implement robust memory monitoring and alerting for your Redis instance.
Solution 3: Scale the Redis Instance
If you regularly approach the memory limit, consider scaling your Redis instance. Increasing the available memory or optimizing Sidekiq job queues can help reduce overall memory consumption and prevent the instance from becoming full.