Terraform Enterprise fails to start on Red Hat Enterprise Linux 8.x

Introduction

This article troubleshoots an issue where the Terraform Enterprise application fails to start on Red Hat Enterprise Linux (RHEL) 8.x and other modern Linux distributions when the restrict_worker_metadata_access setting is enabled.

Problem

The Terraform Enterprise application fails to start because it cannot add a required firewall rule to iptables. The tfe-bootstrap container log displays an error message indicating that a chain, target, or match does not exist.

$ docker logs tfe-bootstrap
[+] Bootstraping TFE docker bridge networks...
network exists tfe_services 8543133ce931089be261ce53c7e1827339c3299869fb1cb663b24af123994db5
network exists tfe_terraform_isolation bd71451a9a98e29f1976ccc3e6008c52eb876ae230c16ebc18e5c50517aeb720
[+] Copying gosu...
'/gosu-bin/gosu' -> '/gosu/gosu'
[+] Managing IMDS firewall rules...
[+] Figuring isolation CIDR block...
[-] Isolation CIDR block is '172.20.0.0/16'
[+] Checking current rules...
[+] Adding network isolation rule...
iptables: No chain/target/match by that name.

Prerequisites

• Terraform Enterprise release v202203-1 (sequence 607) or later.
• Host OS is Red Hat Enterprise Linux 8.x or Amazon Linux 2023.
• The restrict_worker_metadata_access setting is enabled (1).

Cause

The restrict_worker_metadata_access setting is incompatible with Terraform Enterprise deployments on RHEL 8+ systems. This incompatibility is due to a change in iptables management on these operating systems.

This setting is designed to prevent Terraform operations from accessing the instance metadata service, which may contain sensitive data such as IAM credentials.

Solutions

There are two approaches to resolve this issue.

Solution 1: Disable Metadata Access Restriction

The most direct solution is to disable the feature.

1. Set the restrict_worker_metadata_access configuration value to 0 in your Terraform Enterprise settings.

2. Restart the Replicated services to apply the change.

   # replicatedctl app restart

Solution 2: Manually Configure Firewall Rules

If you must keep the metadata restriction enabled, you can manually add the necessary iptables rules to restrict access from the network block where Terraform operation containers are attached.

1. Retrieve the subnet for the Terraform isolation network and store it in a variable. You can use jq or standard Docker commands.

   ## Store the Subnet value in the $cidr variable using jq
   $ cidr=$(curl -s --unix-socket /var/run/docker.sock http://docker/networks/tfe_terraform_isolation \
     | jq -r '.IPAM.Config[].Subnet')
   
   ## Or, if jq is not available, use docker inspect
   $ cidr=$(docker network inspect tfe_terraform_isolation -f '{{range.IPAM.Config}}{{.Subnet}}{{end}}')

2. Create the iptables rule to drop traffic from the Terraform runner containers to the instance metadata service IP address (169.254.169.254).

   # iptables -I DOCKER-USER \
     -s ${cidr} \
     -d 169.254.169.254 \
     -m comment \
     --comment "prevent terraform runners from talking to instance metadata" \
     -j DROP

Outcome

After applying one of the solutions, Terraform Enterprise should start successfully. If you chose Solution 2, the manually added iptables rule in the DOCKER-USER chain will drop traffic originating from Terraform plan and apply containers that is destined for the cloud provider's instance metadata service.

Additional Information

For more details on Terraform Enterprise configuration settings, please refer to the official documentation.