Terraform Enterprise Increased Memory Usage

Problem

You may experience spikes in memory usage that impact Terraform Enterprise (TFE) performance and reliability. Symptoms can include a sluggish user interface and delays in run execution.

Prerequisites

• Affected TFE versions:v202409-3 and below
• Affected deployments: Replicated and Flexible Deployment Options (FDO)

Cause

This issue is caused by a memory leak within the TFE Puma process. The problem is significantly worsened by integrations that frequently access the deprecated api/v2/vars endpoint. High utilization of this endpoint is a major contributor to resource consumption and can lead to out-of-memory (OOM) events.

Solutions

To resolve this issue, you must upgrade Terraform Enterprise and migrate any integrations using the deprecated variables API endpoint.

Solution 1: Upgrade TFE and Migrate API Integrations

Follow these two steps to resolve the performance degradation.

1. Upgrade Terraform Enterprise

   Upgrade your TFE instance to version v202410-1 or newer. This release contains a critical memory leak fix and an upgrade to Ruby 3.3, which introduces further memory and performance improvements.

2. Migrate API Integrations

   Update all existing integrations to stop using the deprecated api/v2/vars endpoint. You should transition to the current workspace variables API endpoint (/workspaces/:workspace_id/vars).

   Note: Upgrading to v202410-1 does not remove access to the deprecated endpoint. Continued use of api/v2/vars may still cause out-of-memory problems until it is removed in a future release.

Outcome

After upgrading to TFE v202410-1 and migrating away from the api/v2/vars endpoint, you should observe a significant reduction in the memory footprint and an improvement in overall platform stability and performance.

Additional Information

• Before upgrading, review the official TFE v202410-1 release notes for a complete list of changes.
• The deprecation bulletin for api/v2/vars provides additional context on this change.