Intro

This article is dedicated to configure proxy settings for Terraform Enterprise Flexible Deployment Options (FDO)

Problem description

In certain environments, the Terraform Enterprise host lacks direct internet access, so a proxy is required for the terraform init stage of runs.

Here's an example of an error message that may appear when Terraform Enterprise is unable to connect to the internet to download the Terraform binary:

Operation failed: failed fetching Terraform: failed downloading terraform: 
failed downloading "https://releases.hashicorp.com/terraform/1.5.3/terraform_1.5.3_linux_amd64.zip": 
GET https://releases.hashicorp.com/terraform/1.5.3/terraform_1.5.3_linux_amd64.zip giving up after 5 attempt(s): 
failed making temp file: open /tmp/terraform/71bf0bd0bd10f6eeac2261e23543df18.download-504ba21b-3a45-c1d4-ff08-7cd03823c4d8: read-only file system

Solutions

Docker

Add proxy environment variables to the docker_compose environment section as described here:

---
version: "3.9"
name: terraform-enterprise
services:
  tfe:
    image: images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202309-1
    environment:
      TFE_HOSTNAME: tfe.domain.com
      http_proxy: http://<your_proxy_server>:8080
      https_proxy: http://<your_proxy_server>:8080
      no_proxy: localhost,127.0.0.1,169.254.169.254,<your_tfe_fqdn>

and restart Terraform Enterprise to apply proxy settings:

sudo docker compose -f docker_compose_file.yml restart

Kubernetes

Add proxy environment variables to the overrides.yaml file of the Terraform Enterprise Helm chart. 

env:
  variables:
    https_proxy: http://<your_proxy_server>:8080
    http_proxy: http://<your_proxy_server>:8080
    no_proxy: localhost,127.0.0.1,169.254.169.254,<your_tfe_fqdn>,<KUBERNETES_SERVICE_HOST>

KUBERNETES_SERVICE_HOST

More details about the KUBERNETES_SERVICE_HOST details can be found here. To get the ip address go to an already running pod on the Kubernetes cluster and verify the environment variable

env | grep KUBERNETES_SERVICE_HOST
KUBERNETES_SERVICE_HOST=172.20.0.1

If you don't add the KUBERNETES_SERVICE_HOST ip address to the no_proxy you could be seeing the following error in the logs when starting a run

{"@level":"error","@message":"error running task instance","@module":"task-worker.executor","@timestamp":"2024-11-12T09:44:20.219907Z",
"err":"error creating kubernetes job: Post \"https://172.20.0.1:443/apis/batch/v1/namespaces/terraform-enterprise-agents/jobs\": context deadline exceeded"}

Agents

The agents that start as part of a workspace run automatically inherit the proxy settings. This is also true when using a custom agent.

env:
  variables:
    https_proxy: http://<your_proxy_server>:8080
    http_proxy: http://<your_proxy_server>:8080
    no_proxy: localhost,127.0.0.1,169.254.169.254,<your_tfe_fqdn>,<KUBERNETES_SERVICE_HOST>
    TFE_RUN_PIPELINE_IMAGE: example/custom-agent:latest