Terraform Cloud Workspace Run Fails with Too Many Open Files

Problem

An HCP Terraform workspace run fails during the plan or apply phase. The run log reports a too many open files error.

Invalid value for "path" parameter: open
...
too many open files.

Prerequisites

• The HCP Terraform workspace Execution Mode is set to remote.

Cause

When a workspace uses the remote execution mode, runs execute in a default HCP Terraform worker container with finite CPU and memory resources. A large or complex configuration can create a high number of concurrent processes, exceeding the resource limits of this container and causing the run to fail.

Solutions

There are two options to resolve this issue. The first solution limits the concurrency of the run to stay within the default worker's resource limits. The second solution uses a self-hosted agent with more flexible resource allocation.

Solution 1: Limit Concurrency with an Environment Variable

You can limit the number of concurrent processes by setting the TFE_PARALLELISM environment variable in the workspace variables.

Note: The TFE_PARALLELISM environment variable controls the number of concurrent operations that HCP Terraform executes. The default value is 10. For more information, refer to the documentation on special environment variables.

1. Navigate to your workspace's Variables page at https://app.terraform.io/app/<ORGANIZATION_NAME>/workspaces/<WORKSPACE_NAME>/variables.
2. Under Environment Variables, select Add variable.
3. Set the Key to TFE_PARALLELISM and the Value to 1.
4. Save the variable and queue a new run to verify that the error is resolved.

Note: For large configurations, decreasing the value to 1 may cause the plan phase to time out. If you encounter a plan timeout, incrementally increase the value of TFE_PARALLELISM until you find the optimal setting for your configuration.

Solution 2: Use Self-Hosted Agents for Increased Resources

If your organization has deployed HCP Terraform Agents, you can change the workspace's execution mode to Agent. Agents run on user-provisioned infrastructure, allowing for adjustable resource allocation that can accommodate larger configurations.

1. Navigate to your workspace's general settings page at https://app.terraform.io/app/<ORGANIZATION_NAME>/workspaces/<WORKSPACE_NAME>/settings/general.
2. Under the Execution Mode setting, select Agent.
3. Scroll to the bottom of the page and select Save settings.
4. Queue a new run and verify its success in the run log.