Terraform Enterprise Restore Returns 500 Error Due to Temporary Storage Limitations – HashiCorp Help Center

Overview

When restoring a backup in Terraform Enterprise (Flexible Deployment Option – Podman/Kubernetes style deployments), the restore process may fail with a 500 Internal Server Error. This typically occurs when the temporary filesystem used by TFE does not have enough space to hold the uploaded snapshot.

html> <head><title>500 Internal Server Error</title></head> <body> <center><h1>500 Internal Server Error</h1></center> <hr><center>nginx</center> </body> </html>

In the Terraform Enterprise logs, the following error appears:

pwrite() "/var/run/terraform-enterprise/tmp/..." failed (28: No space left on device)

Nginx logs may also show:

POST /_backup/api/v1/restore HTTP/1.1" 500

When This Occurs

This issue commonly occurs when:

Performing a restore operation on Terraform Enterprise
The backup snapshot is large (several GBs or more)

Root Cause

During the restore process, Terraform Enterprise temporarily writes the uploaded snapshot to:

/var/run/terraform-enterprise/tmp

If this path resides on a memory-backed filesystem, large backup files cannot be fully written, resulting in:

No space left on device

Even if the host has sufficient disk space, the restore will still fail because the temporary write location is memory-based.

Solution

To restore large backups successfully, temporarily redirect the restore temporary directory to a disk-backed location.

Recommended Approach

Mount a host directory with sufficient free space to:

/var/run/terraform-enterprise/tmp

Implementation Steps

1. Create a directory on the host

Example:

mkdir -p /opt/terraform/restore-tmp
chmod 777 /opt/terraform/restore-tmp

Ensure the filesystem has enough free space for the snapshot.

2. Modify the Terraform Enterprise configuration

Add a temporary volume mount.

Example:

volumeMounts:
  - mountPath: "/run"
    name: "run"
  - mountPath: "/tmp"
    name: "tmp"
  - mountPath: "/var/run/terraform-enterprise/tmp"
    name: "temporary-restore"

volumes:
  - emptyDir:
      medium: "Memory"
    name: "run"

  - emptyDir:
      medium: "Memory"
    name: "tmp"

  - hostPath:
      path: "/opt/terraform/restore-tmp"
      type: "Directory"
    name: "temporary-restore"

3. Restart Terraform Enterprise

Apply the configuration changes and restart TFE.

4. Run the Restore Again

Re-run the restore command.

Best Practices

Ensure the temporary restore location has at least the size of the backup available, preferably 1.5× the backup size.
Avoid using memory-backed volumes for large restore operations.
Keep /run and /tmp as memory-backed volumes for normal operation, but override only the restore temp path when needed.

References -