Terraform plan fails with tfe-agent failed to load image

Problem

A Terraform plan in Terraform Enterprise fails because the task-worker cannot pull the hashicorp/tfe-agent image. The logs show a "permission denied" error during driver initialization.

Error: failed pulling image "hashicorp/tfe-agent:now": Error response from daemon: {"message":"denied: requested access to the resource is denied"}

Cause

There are two primary causes for this issue:

1. Invalid License: The Terraform Enterprise license is invalid or misconfigured, which prevents authenticated access to the HashiCorp image registry.
2. Insufficient Disk Space: The host machine has run out of disk space in the directory used by the container runtime (e.g., Podman or Docker) for image storage, typically /var.

Solutions

Follow these solutions to diagnose and resolve the image pull failure.

Solution 1: Validate the Terraform Enterprise License

An invalid license can cause authentication to the image registry to fail. You can validate your license with the following commands.

1. Log in to the HashiCorp image registry using your license key.

   $ echo "<HASHICORP_LICENSE>" | docker login --username terraform images.releases.hashicorp.com --password-stdin

2. Attempt to pull the agent image directly.

   $ docker --debug pull hashicorp/tfe-agent:latest

If these commands fail, your license may be invalid. Contact your HashiCorp representative to obtain a valid license.

Solution 2: Increase Disk Space

Insufficient disk space is the most common cause of this error. You may see a related error in the logs indicating no space is left on the device.

Error response from daemon: failed to load image: payload does not match any of the supported image formats:
* oci: open /var/tmp/api_load.tarXXXX/index.json: not a directory
* oci-archive: creating temp directory: untarring file "/var/tmp/container_images_ociXXXX":
write ... no space left on device

1. Increase the size of the /var partition to a minimum of 40GB. This is the recommended size for stable Terraform Enterprise operations.
2. Ensure that the specific storage directory for your container runtime, such as /var/lib/containers/storage for Podman, has sufficient space available.

3. After resizing the partition, redeploy Terraform Enterprise. The following example uses Podman.

   ## Bring down the current deployment
   $ podman kube down /path/to/your/manifest.yaml
   
   ## Redeploy using the manifest
   $ podman play kube /path/to/your/manifest.yaml

Additional Information

• HCP Terraform Agents documentation