Introduction
This article explains a scenario where Terraform runs fails when TFE is deployed on OpenShift. The error typically occurs when Terraform Enterprise and TFE Agents are deployed within the same OpenShift namespace during parallel runs.
Problem
Terraform runs fail with a module download error when attempting to retrieve modules from Terraform Enterprise.
Example error:
Error: Failed to download module
on compute.tf line 714:
714: module "compute" {
Could not download module "compute" (compute.tf:714) source code
from "https://terraformenterprise.domain.com/_archivist/v1/object/... ?archive=tgz":
bad response code: 503This issue is observed when Terraform Enterprise is deployed on OpenShift and multiple runs are executed in parallel.
Prerequisites (if applicable)
This article applies to the following environments:
Terraform Enterprise deployed on OpenShift Terraform runs executed using Terraform Enterprise Agents Parallel Terraform runs retrieving modules from TFE Terraform Enterprise and TFE Agents deployed in the same namespace
Cause
This issue occurs when Terraform Enterprise is deployed in OpenShift.
The environment variable TFE_RUN_PIPELINE_KUBERNETES_NAMESPACE is configured to specify the Terraform Enterprise Agent namespace, and it is set to the same namespace where Terraform Enterprise is deployed, which can lead to run failures and 503 Service Unavailable errors during parallel runs.
This configuration is not supported.
Terraform Enterprise Agents are expected to run in a separate namespace from the Terraform Enterprise deployment to ensure proper network isolation and stable service communication.
Overview of possible solutions
To resolve this issue, ensure that Terraform Enterprise Agents are deployed in a different namespace from Terraform Enterprise.
Solutions
Solution 1: Deploy TFE Agents in a Separate Namespace (Recommended)
Create a dedicated namespace for Terraform Enterprise Agents instead of using the same namespace as Terraform Enterprise.
Example:
Create a new namespace for agents:
oc create namespace tfe-agents
Deploy the Terraform Enterprise Agents into this namespace using TFE_RUN_PIPELINE_KUBERNETES_NAMESPACE Ensure the agents are properly configured to connect to the Terraform Enterprise instance. Re-run the Terraform workspace runs.
This configuration aligns with the supported architecture for Terraform Enterprise Agents.
Solution 2: Run Terraform Jobs Sequentially (Temporary Mitigation)
If separating namespaces is not immediately possible, run Terraform jobs one at a time to avoid parallel execution.
This can be done by:
Option A: Trigger runs manually one by one so that only a single run executes at a time.
Option B: Limit Terraform Enterprise run concurrency by configuring the following environment variable:
TFE_CAPACITY_CONCURRENCY=1
Setting this value ensures that only one Terraform run executes at a time, preventing the parallel requests that can trigger the 503 error.
Note:
This is only a temporary mitigation and does not resolve the unsupported configuration. The recommended approach remains deploying TFE Agents in a separate namespace.
Outcome
After deploying Terraform Enterprise Agents in a separate namespace:
Terraform runs should successfully download modules. The 503 error during module download should no longer occur. Parallel runs should execute without module retrieval failures.