Terraform Enterprise fails to start on Red Hat Enterprise Linux 8.x

Introduction

Problem

The Terraform Enterprise application fails to start when restrict_worker_metadata_access is enabled due to being unable to add a firewall rule to iptables.

root@ip-10-0-40-36:~# docker logs tfe-bootstrap
[+] Bootstraping TFE docker bridge networks...
2023/11/03 18:20:27 network exists tfe_services 8543133ce931089be261ce53c7e1827339c3299869fb1cb663b24af123994db5
2023/11/03 18:20:27 network exists tfe_terraform_isolation bd71451a9a98e29f1976ccc3e6008c52eb876ae230c16ebc18e5c50517aeb720
[+] Copying gosu...
'/gosu-bin/gosu' -> '/gosu/gosu'
[+] Managing IMDS firewall rules...
[+] Figuring isolation CIDR block...
[-] Isolation CIDR block is '172.20.0.0/16'
[+] Checking current rules...
[+] Adding network isolation rule...
iptables: No chain/target/match by that name.

Prerequisites

• Terraform Enterprise release sequence 607 and later
• Red Hat Enterprise Linux 8.x
• restrict_worker_metadata_access is set to 1

Cause

• The restrict_worker_metadata_access setting is incompatible with TFE deployments on RHEL 8+ systems, due to a change in the iptables management systems present in RHEL 8 systems.
• restrict_worker_metadata_access is used to prevent Terraform operations from accessing the instance metadata service, which may contain IAM credentials or other sensitive data.

Solutions:

• Disable the restrict_worker_metadata_access configuration by setting the value to "0".

• Start or restart Replicated:

# replicatedctl app restart

• Manually add the necessary iptables rules to restrict access from the network block where Terraform operation containers are attached:

## Store the Subnet value in the $cidr variable

# cidr=$(curl -s --unix-socket /var/run/docker.sock http://docker/networks/tfe_terraform_isolation \
   | jq -r '.IPAM.Config[].Subnet')

## Or if jq is not available

# cidr=$(docker network inspect tfe_terraform_isolation -f '{{range.IPAM.Config}}{{.Subnet}}{{end}}')

## Create Iptables rule

# iptables -I DOCKER-USER \
  -s ${cidr} \
  -d 169.254.169.254 \
  -m comment \
  --comment "prevent terraform runners from talking to instance metadata" \
  -j DROP

Outcome

Once the above iptables rule has been added to the DOCKER-USER chain, traffic destined toward the cloud service provider's instance metadata service will be dropped, when that traffic originates from Docker containers used for Terraform plan/apply operations.