Terraform Enterprise FDO Podman: failed fetching Terraform ... read-only file system

Problem

During a Terraform run in a Terraform Enterprise Flexible Deployment Options (FDO) environment using Podman, you see an error indicating a read-only file system.

[ERROR] terraform: Failed handling run: error="operation failed: failed fetching Terraform: failed downloading terraform: failed downloading "https://releases.hashicorp.com/terraform/1.3.5/terraform_1.3.5_linux_amd64.zip": GET "https://releases.hashicorp.com/terraform/1.3.5/terraform_1.3.5_linux_amd64.zip": giving up after 5 attempt(s): failed making temp file: open /tmp/terraform/aefsdafdsfasfsadfasfasfafaf8d5.download-f8fsdfdsfasdfsafasdfasffa4: read-only file system

Prerequisites

• Terraform Enterprise installed with Flexible Deployment Options (FDO) for Podman.

Cause

This error typically occurs when the tfe.yaml configuration file for Podman has misconfigured settings related to the disk cache volume. Common causes include:

• The environment variable TFE_DISK_CACHE_VOLUME_NAME is missing or does not point to a valid volume.

• The volume mount for the cache directory is missing or misconfigured. It should appear similar to the following snippet.

  - mountPath: "/var/cache/tfe-task-worker/terraform"
    name: "terraform-enterprise_terraform-enterprise-cache-pvc"

• The volume definition is missing or incorrect. It should appear similar to the following snippet.

  - name: "terraform-enterprise_terraform-enterprise-cache-pvc"
    persistentVolumeClaim:
      claimName: "terraform-enterprise_terraform-enterprise-cache"

Solution

Verify that your tfe.yaml file matches the official example configurations from the Deploy Terraform Enterprise to Podman documentation.

Pay close attention to the sections defining the disk cache environment variable, volume mount, and volume. The following example shows the correct structure for these settings.

---
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    app: "terraform-enterprise"
  name: "terraform-enterprise"
spec:
  restartPolicy: "Never"
  containers:
    - env:
        - name: "TFE_DISK_CACHE_VOLUME_NAME"
          value: "terraform-enterprise_terraform-enterprise-cache"
      # ... other environment variables
      volumeMounts:
        - mountPath: "/var/cache/tfe-task-worker/terraform"
          name: "terraform-enterprise_terraform-enterprise-cache-pvc"
  # ... other spec configurations
  volumes:
    - name: "terraform-enterprise_terraform-enterprise-cache-pvc"
      persistentVolumeClaim:
        claimName: "terraform-enterprise_terraform-enterprise-cache"

Outcome

After correcting the tfe.yaml configuration and restarting Terraform Enterprise, the file system error should be resolved, and Terraform runs should complete successfully.