Terraform Download Fails In Remote Runs Due to Security Agent

Introduction

This article addresses a run failure in Terraform Enterprise where the remote worker cannot download the required Terraform binary due to interference from a security agent.

Problem

Terraform runs are failing with the following error, indicating the remote Terraform worker (Terraform Cloud Agent) is unable to download the Terraform archive for use in the run.

Operation failed: failed fetching Terraform: failed downloading terraform: failed downloading "https://releases.hashicorp.com/terraform/1.6.6/terraform_1.6.6_linux_amd64.zip": GET https://releases.hashicorp.com/terraform/1.6.6/terraform_1.6.6_linux_amd64.zip giving up after 5 attempt(s): failed making temp file: open /tmp/terraform/73dd7d82855be55a5f64658eec0c7e5d.download-e29388b4-a5c5-dd53-795d-5e92e330f239: read-only file system

Prerequisites

• Terraform Enterprise >= v202302-1
• run_pipeline_mode is set to agent (only v202302-1 through v202305-1)
• Workspace execution mode is set to remote
• Security agent(s) installed on the system (e.g., CrowdStrike Falcon Sensor)

Cause

When a run is triggered, an agent job is sent to the task-worker service, which starts an ephemeral Terraform worker to perform the run. As a pre-task, the task-worker downloads the Terraform archive required by the Terraform Enterprise workspace to a shared cache. This cache is then made available to the ephemeral agent to extract and use during the run.

The download is performed by an external program called by the task-worker process, which is located at /usr/bin/tfe-task-worker-downloader (Replicated deployment type in non-consolidated services mode) or /usr/local/bin/tfe-task-worker-downloader (Replicated in consolidated services and FDO).

In some Terraform Enterprise environments, security agents such as the CrowdStrike Falcon Agent may block the tfe-task-worker-downloader program, preventing the Terraform binary from being downloaded to the cache. The error occurs when the Terraform worker attempts to download the missing Terraform archive to its own read-only filesystem, an action that should have already been performed by the task-worker service.

Solutions

Solution 1: Allowlist the Downloader Program

Contact your organization's CrowdStrike administrator to allowlist the tfe-task-worker-downloader file to resolve this false positive.

Outcome

Once the downloader program is allowlisted, plans and applies that use a Terraform version not already present in the cache will complete successfully.

Additional Information

For more details on Terraform Enterprise architecture, please refer to the official documentation regarding agents and operational security.