Problem
There is a known issue where customers may experience spikes in memory usage impacting Terraform Enterprise (TFE) performance and reliability. The UI may be sluggish to navigate and operate and runs could be delayed in execution.
Prerequisites
- Affected TFE versions are v202409-3 and below
- Affects both TFE Replicated and FDO deployments
Cause
- Memory utilization within the Puma process
- Utilizing the api/v2/vars endpoint
- This endpoint has been deprecated and it is recommended to discontinue use
- Accessing the api/v2/vars endpoint is a major resource usage contributor and it is expected to cause out of memory (OOM) scenarios
Solutions:
- If the TFE system is experiencing these issues, it is recommended to upgrade TFE to v202410-1
-
TFE version v202410-1 contains a vital memory leak fix
-
This fix is expected to lower the overall memory footprint
-
There is also a upgrade to Ruby 3.3 which introduces memory and performance improvements
-
-
Please note this release does not remove access to the API deprecated endpoint api/v2/vars
- Reaching this endpoint after upgrading to v202410-1 may still cause OOM problems
- It is recommended to cease utilizing the api/v2/vars endpoint until it is disabled entirely in a future release
-
All existing integrations with this API should transition to the workspace variables API /workspaces/:workspace_id/vars.
Outcome:
- Upgrading to v202410-1 along with not utilizing the API api/v2/vars (if applicable) endpoint should significantly improve and reduce the memory footprint
Additional Information
-
Best practices when considering upgrading is to review the release notes for all changes that may impact the platform
- api/v2/vars deprecation bulletin