Runs fail with "Setup failed: Failed setting up Terraform binary: Failed pushing binary to environment: exit status 125" – HashiCorp Help Center

Problem

During the plan phase of a run in Terraform Enterprise, the run fails with the following error message in the plan output:

Setup failed: Failed setting up Terraform binary: Failed pushing binary to environment: exit status 125

Prerequisites

Terraform Enterprise versions prior to v202306-1.

Cause

Terraform Enterprise starts a disposable Docker container to perform the plan or apply phase of a run. The image this container uses is based on the configured worker image. If that image does not exist on the instance, the container cannot start, causing the run to fail.

This issue can occur if the default worker image was removed from the instance, for example, by running a docker prune command. If you use an alternative worker image, the image may not be available on the instances where Terraform Enterprise is running.

To confirm this is the cause, check the logs for the ptfe_build_worker container (or tfe-build-worker for Terraform Enterprise v202205-1 or later) for a log entry similar to the following.

{
  "@level": "error",
  "@message": "(Docker: <uuid>) Failed to start container: exit status 125\nOutput:\nUnable to find image '<image name>' locally\ndocker: Error response from daemon: pull access denied for worker, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.\nSee 'docker run --help'.",
  "@module": "terraform-build-worker.stdlog",
  "@timestamp": "2021-11-07T22:44:13.486100Z",
  "git_commit": "0e37e08",
  "isolation_type": "docker"
}

Unless configured otherwise, Terraform Enterprise uses a default worker image. To determine which image is in use, check the configured value for tbw_image.

$ replicatedctl app-config export --template "{{.tbw_image.Value}}"

Valid values are default_image or custom_image.

If the value is default_image, the image name used is hashicorp/build-worker:now.
If the value is custom_image, the image name is the value configured for custom_image_tag.

Solutions

There are two solutions depending on whether you use the default or a custom worker image.

Solution 1: Restart the Application to Recreate the Default Worker Image

If the default worker image was accidentally removed, restarting the Terraform Enterprise application will recreate it.

Stop the application.
```
$ replicatedctl app stop
```
Wait for the application to stop. Check the status until the State transitions to stopped.
```
$ replicatedctl app status
```
Start the application.
```
$ replicatedctl app start
```

Solution 2: Manually Pull the Alternative Worker Image

If you use an alternative worker image, you must pull the image onto the instance. A restart of the application is not required.

For Active/Active installations, you should pull the alternative worker image as part of the node creation process to ensure new nodes can perform runs as they come online.

Outcome Validation

Once the worker image has been restored on the instance, subsequent plans should complete successfully.

Additional Information