Terraform runs don't execute in Terraform Enterprise

Avatar
Patrick Munne
  • Updated

Problem

Terraform runs suddenly don't execute as expected. Terraform runs are executed by starting a docker container named tfe-build-worker. Inspecting the containers logs for the following error messages will confirm if this issue affects your Terraform Enterprise instance.

  • Run this command to verify
docker logs tfe-build-worker
  • output example
2023-02-03T08:49:33.062759000Z ------
2023-02-03T08:49:33.062829000Z > [internal] load metadata for docker.io/library/94818adc3a27:latest:
2023-02-03T08:49:33.062897000Z ------
2023-02-03T08:49:33.062952000Z 94818adc3a27: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
2023-02-03T08:49:34.525469000Z #1 [internal] load .dockerignore
2023-02-03T08:49:34.525886000Z #1 sha256:debf8ad19e2cff8fba9518a13c11d05a994f152db0e519fd9d7a8813a6b59671
2023-02-03T08:49:34.674101000Z #1 transferring context: 2B done
2023-02-03T08:49:34.674345000Z #1 DONE 0.0s
2023-02-03T08:49:34.674493000Z #2 [internal] load build definition from Dockerfile
2023-02-03T08:49:34.674555000Z #2 sha256:c71faff7c8ae71c08fc6f734fd105f95d039552e66b6360cf43aa8ec93a67119
2023-02-03T08:49:34.674611000Z #2 transferring dockerfile: 490B 0.0s done
2023-02-03T08:49:34.674669000Z #2 DONE 0.0s
2023-02-03T08:49:34.674779000Z #3 [internal] load metadata for docker.io/library/94818adc3a27:latest
2023-02-03T08:49:34.674837000Z #3 sha256:8762cc4438dec6cd2d8ec9e9ca3369810268d3e5c7fca157ab08d01d098d7a95
2023-02-03T08:49:35.091218000Z #3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
2023-02-03T08:49:35.091400000Z ------

 

Cause

  • The Docker Engine version is not supported for Terraform Enterprise. 
  • For example, If you recently upgraded to Docker Engine 23.0 or Docker Engine 24.0, this is currently not a supported version of Docker for releases older then v202306-1 (713)
docker info | grep "Server Version"
Server Version: 23.0.0                <-- currently not supported

Solutions:

  1. Install the docker version that is supported before installing Terraform Enterprise
         - See our supported docker versions here

  2. When using the Terraform online installation the installation script will automatically install docker version 24.x. To install an older version please use the following flag during the installation
./install.sh docker-version=20.10.17

Supported docker version output:

docker info | grep "Server Version"
Server Version: 20.10.17
  • Modifying the /etc/docker/daemon.json Docker daemon configuration file with specific settings can restore the previous functionality when using Docker version 23+. However, it's important to note that this configuration is not officially supported and could lead to unexpected behavior or issues. It's always recommended to use supported configurations for better stability and security. If any issues are encountered or have questions about the compatibility of Terraform Enterprise with Docker 23+, please reach out to the Terraform Enterprise support team for assistance.
    /etc/docker/daemon.json
    {
      "features": {
      "buildkit" : false
     }
    }

    # Stop the TFE app if still running
    replicatedctl app stop

    # Stop Replicated services
    sudo systemctl stop replicated replicated-operator replicated-ui

    # Stop Docker service
    sudo systemctl stop docker

    # Reload the Systemd Daemon
    sudo systemctl daemon-reload

    # Start Docker service
    sudo systemctl start docker

    # Verify Replicated has started successfully before starting the TFE app
    replicatedctl system status
    {
    "Replicated": "ready",
    "Retraced": "ready"
    }

    # Start the TFE app
    replicatedctl app start

Additional Information

Articles in this section

See more

Related articles