Problem
When the plan phase of a run starts, the run is errored and the plan output is:
Setup failed: Failed setting up Terraform binary: Failed pushing binary to environment: exit status 125
Prerequisites
- Terraform Enterprise, prior to v202306-1
Cause
When the plan or apply phase of a run is performed, a disposable Docker container is started to perform the action. The image this container will use is based on the configured worker image. If that image doesn't exist the container will be unable to start.
This can occur if the default worker image has been removed from the instance, possibly removed when performing a Docker prune. If an alternative worker image is in use, the image may not be available on the instances Terraform Enterprise is running on.
To confirm this issue is occuring, check the logs for the container ptfe_build_worker or tfe-build-worker(Terraform Enterprise v202205-1 or later) looking for a log entry similar to this:
{"@level":"error","@message":"(Docker: <uuid>) Failed to start container: exit status 125\nOutput:\nUnable to find image '<image name>' locally\ndocker: Error response from daemon: pull access denied for worker, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.\nSee 'docker run --help'.","@module":"terraform-build-worker.stdlog","@timestamp":"2021-11-07T22:44:13.486100Z","git_commit":"0e37e08","isolation_type":"docker"}
Unless configured to, Terraform Enterprise will use a default worker image. To determine which is in use, check the configured value for tbw_image with:
replicatedctl app-config export --template "{{.tbw_image.Value}}"
Valid values are either default_image or custom_image.
If it is default_image, the image name used will be hashicorp/build-worker:now.
If it is custom_image, the image name will be the value configured for custom_image_tag.
Solutions
If the default worker image has been accidentally removed, restarting the application will recreate it:
replicatedctl app stop- Wait for the application to stop, check with
replicatedctl app statusuntil "State" transitions to "stopped" replicatedctl app start
If an alternative worker image is configured, pull the image on the instance. Once the image is pulled, runs should start working. Restarting the application isn't required. If you have an Active/Active installation, the alternative worker image should be pulled as part of node creation to ensure it's able to perform runs as it comes online.
Outcome
Once the worker image has been restored, subsequent plans should complete successfully. If the issue persists and no rules are being triggered, please reach out to HashiCorp support for additional assistance.