Understanding Redis Eviction Behavior with Sidekiq and Terraform Enterprise (TFE) – HashiCorp Help Center

Introduction

This article explains the expected Redis behavior when used with Sidekiq in a Terraform Enterprise (TFE) environment. It also clarifies key management expectations and the recommended Redis configuration to prevent unintended key evictions.

Issue

A customer observed Redis evicting keys upon reaching the maxmemory threshold. While this does not cause application crashes, it raises concerns about potential unintended behavior and the need to better understand Redis and TFE’s handling of key eviction.

Cause

Redis Evictions:
When Redis reaches its maxmemory limit, it begins evicting keys based on its configured eviction policy (e.g., volatile-lru, allkeys-lru, etc.). This can lead to unexpected loss of data if not configured properly for applications like Sidekiq.
Sidekiq's Expectations:
Sidekiq is designed to manage its own keys in Redis and expects no eviction behavior from the Redis instance. Any eviction undermines Sidekiq’s internal key lifecycle management and can result in missing or inconsistent job state.
TFE & Rails Key Management:
Terraform Enterprise and its underlying Rails application do not perform manual Redis key management. Redis usage is largely handled by Sidekiq, and thus should follow Sidekiq’s configuration guidance

Solutions

Set Redis Eviction Policy to noeviction:
This gives Sidekiq full control over key management.
Example Redis config:

maxmemory 2gb
maxmemory-policy noeviction

Monitor Memory Usage:
Since noeviction can lead to Redis refusing writes when memory is exhausted, monitoring tools and memory alerts are recommended.
Scale Redis Appropriately:
If memory limits are being hit regularly, consider increasing the Redis instance size or optimizing Sidekiq job queues to reduce memory consumption.
To ensure stability and predictable behavior in environments using Sidekiq with TFE:

Key eviction in Redis when using Sidekiq is not expected behavior and can interfere with job processing. The Redis instance used with Terraform Enterprise should be configured with noeviction to align with Sidekiq’s operational model.

Additional Information