Introduction
Problem
The Terraform Enterprise application fails to start when restrict_worker_metadata_access is enabled due to being unable to add a firewall rule to iptables.
Prerequisites
- Terraform Enterprise release sequence 607 and later
- Red Hat Enterprise Linux 8.x
- restrict_worker_metadata_access is set to 1
Cause
- The restrict_worker_metadata_access setting is incompatible with TFE deployments on RHEL 8+ systems, due to a change in the iptables management systems present in RHEL 8 systems.
- restrict_worker_metadata_access is used to prevent Terraform operations from accessing the instance metadata service, which may contain IAM credentials or other sensitive data.
Solutions:
-
Disable the restrict_worker_metadata_access configuration by setting the value to "0".
- Start or restart Replicated:
# replicatedctl app restart
- Manually add the necessary iptables rules to restrict access from the network block where Terraform operation containers are attached:
## Store the Subnet value in the $cidr variable
# cidr=$(curl -s --unix-socket /var/run/docker.sock http://docker/networks/tfe_terraform_isolation \
| jq -r '.IPAM.Config[].Subnet')
## Or if jq is not available
# cidr=$(docker network inspect tfe_terraform_isolation -f '{{range.IPAM.Config}}{{.Subnet}}{{end}}')
## Create Iptables rule
# iptables -I DOCKER-USER \
-s ${cidr} \
-d 169.254.169.254 \
-m comment \
--comment "prevent terraform runners from talking to instance metadata" \
-j DROP
Outcome
Once the above iptables rule has been added to the DOCKER-USER chain, traffic destined toward the cloud service provider's instance metadata service will be dropped, when that traffic originates from Docker containers used for Terraform plan/apply operations.