Terraform Download Fails In Remote Runs Due to Security Agent

Problem

Terraform runs are failing with the following error, indicating the remote Terraform worker (Terraform Cloud Agent) is unable to download the Terraform archive for use in the run. 

Operation failed: failed fetching Terraform: failed downloading terraform: failed downloading "https://releases.hashicorp.com/terraform/1.6.6/terraform_1.6.6_linux_amd64.zip": GET https://releases.hashicorp.com/terraform/1.6.6/terraform_1.6.6_linux_amd64.zip giving up after 5 attempt(s): failed making temp file: open /tmp/terraform/73dd7d82855be55a5f64658eec0c7e5d.download-e29388b4-a5c5-dd53-795d-5e92e330f239: read-only file system

Prerequisites 

• Terraform Enterprise >= v202302-1
• run_pipeline_mode is set to agent (only v202302-1 through v202305-1)
• Workspace execution mode is set to remote
• Security agent(s) installed on system (i.e CrowdStrike Falcon Sensor)

Cause

When a run is triggered, an agent job is sent to the task-worker service, which it will start an ephemeral Terraform worker to perform the run. As a pre-task of this job, the task-worker downloads the Terraform archive required by the Terraform Enterprise workspace to a shared cache which is made available to the ephemeral agent to extract and utilize during the run. The download is performed by an external program called by the task-worker process which is located at /usr/bin/tfe-task-worker-downloader (Replicated deployment type in non-consolidated services mode) or /usr/local/bin/tfe-task-worker-downloader (Replicated in consolidated services and FDO).

In Terraform Enterprise, security agents such as CrowdStrike Falcon Agent have been known to block the tfe-task-worker-downloader program, preventing the Terraform binary from being downloaded to the cache prior to the run. The error is thrown by the Terraform worker as it attempts to download the non-present Terraform archive to its (read-only) filesystem, an action would should have already been performed by the task-worker service.

Solutions:

Contact your organization's CrowdStrike administrator to allowlist the tfe-task-worker-downloader file to resolve this false positive.

Outcome

Once resolved, plan and applies which use a Terraform version not already present in the cache complete successfully.