Terraform Enterprise Fails to Start With "Read-only file system" Error In tfe-bootstrap Container (Replicated) – HashiCorp Help Center

Problem

Terraform Enterprise fails to start, and the replicatedctl app status command shows the application in a stopped state.

[
  {
    "AppID": "4b97c9a63fe44edb4cd16ca99d97da59",
    "Sequence": 776,
    "PatchSequence": 0,
    "State": "stopped",
    "DesiredState": "started",
    "Error": "Container tfe-bootstrap failed: Container 527a3d6bb094d9e7dd0f22264496bd59cb5b57af162cef77a3a95e3eae1a1e43 exited with non-zero exit status 4: ",
    "IsCancellable": false,
    "IsTransitioning": false,
    "LastModifiedAt": "2024-07-01T16:32:21.152313367Z"
  }
]

Reviewing the logs for the tfe-bootstrap container reveals a fatal error related to a read-only file system.

Fatal: can't open lock file /run/xtables.lock: Read-only file system

Prerequisites

Terraform Enterprise v202407-1 or later (Replicated deployment).
The restrict_worker_metadata_access setting is enabled in the Terraform Enterprise configuration.

Cause

When you enable the restrict_worker_metadata_access setting, Terraform Enterprise starts an ephemeral container, tfe-bootstrap, to create an iptables rule. This rule prevents containers on the tfe_terraform_isolation Docker network from accessing the instance metadata endpoint at 169.254.169.254.

A change introduced in Terraform Enterprise v202305-1 (Replicated deployment) modifies the filesystem properties of the tfe-bootstrap container. This change can cause the iptables command to fail when it cannot obtain a lock, resulting in the error.

Solution

This issue is a known bug that is being addressed. The following workaround allows you to manually create the required iptables rule.

Workaround: Manually Create the iptables Rule

Disable the restrict_worker_metadata_access setting in your Terraform Enterprise configuration to prevent the tfe-bootstrap container from attempting to create the rule.
Apply the configuration and restart Terraform Enterprise.

After the application is running, manually create the iptables rule on the host machine. The following command retrieves the Docker network subnet and adds the rule to block access to the instance metadata endpoint.

Execute the command as a privileged user.

# cidr=$(docker network inspect tfe_terraform_isolation -f '{{range.IPAM.Config}}{{.Subnet}}{{end}}') && \
  iptables -I DOCKER-USER \
  -s ${cidr} \
  -d 169.254.169.254 \
  -m comment \
  --comment "prevent terraform runners from talking to instance metadata" \
  -j DROP

Additional Information