How to Block Remote Worker Access to Instance Metadata Endpoint in Terraform Enterprise Flexible Deployment Options

Overview

By default, Terraform Enterprise does not prevent remote workers from accessing the Instance Metadata Service (IMDS). In the legacy Replicated deployment, access to IMDS from remote workers could be blocked by setting restrict_worker_metadata_access in the application settings file. This setting effectively created an iptables rule on the server which disallowed traffic to 169.254.169.254 from all source IPs on the tfe_terraform_isolation Docker bridge network, a Docker network created and managed by Terraform Enterprise for remote agent containers.
 
This setting is no longer available in Flexible Deployment Options (FDO), given its multi-platform support. Now, Terraform Enterprise (TFE) provides users with the configuration primitives required to block this access themselves. HashiCorp recommends blocking Terraform operations from accessing the instance metadata endpoint, unless workspaces rely on the instance profile to provide default credentials to workspaces.

Procedure

The general steps required to configure block remote workers' access to IMDS in a FDO Docker deployment are as follows.

Docker/Podman

• Create a user-defined Docker bridge network (see these instructions for Podman) on each TFE node for the remote agent containers
• Add the TFE_RUN_PIPELINE_DOCKER_NETWORK option to the TFE configuration with the name of the user-defined bridge network as its value.

• Create an iptables rule which blocks traffic to IMDS from any source IPs on this network.
   

  cidr=$(docker network inspect <CUSTOM_DOCKER_NETWORK_NAME> -f '{{range.IPAM.Config}}{{.Subnet}}{{end}}')
  
  iptables -I DOCKER-USER \ 
    -s ${cidr} \
    -d 169.254.169.254 \
    -m comment \
    --comment "block terraform workers from connecting to IMDS" \
    -j DROP

Kubernetes

In Kubenetes, consider implementing one of the following solutions.

• If using Cloud Provider-managed Kubernetes, enable any available cluster settings to block access (i.e the --enable-imds-restriction setting in AKS).
• Configure a Network Policy which blocks access to the IMDS endpoint from the agents namespace.

Note that any workspaces which rely on sourcing AWS credentials from IMDS will no longer be able to do so after implementing this and will need to be configured with an alternate authentication method.

Additional Information

• Restrict Terraform Build Worker Metadata Access
• Terraform Enterprise configuration reference
• Refer to AWS, Azure, or Google Cloud documentation for more information on the instance metadata service.