[TFE] Docker fails to pull images post re-deployment

Introduction

This article troubleshoots an issue where a Terraform Enterprise installation using Replicated fails to start because Docker is unable to pull the necessary images.

Problem

During the installation or restart of Terraform Enterprise, the application does not become available. The Replicated admin console displays an error.

Running the replicatedctl app status command shows the following error, even with a valid license installed.

$ replicatedctl app status
Error: App not found. It is possible that you have not yet uploaded a license. Please first install a license using the license-load command.

Docker logs indicate an image pull failure with messages such as unauthorized or image not found.

level=error msg="Not continuing with pull after error: error: image replicated/replicated-operator:stable-2.56.6 not found"
level=error msg="Attempting next endpoint for pull aftererror: unauthorized: access to the requested resource is not authorized"
level=error msg="Not continuing with pull after error: Error: image replicated/replicated-operator:stable-2.56.6 not found"
ERROR daemon/daemon.go:1443 Support bundle pull failed with error: received unexpected HTTP status: 503 Service

Logs in /var/log/messages may show a 503 Service Unavailable error.

error msg-“Download failed retrying：received unexpected HTTP status: 503 service Unavailable

Verifying the local Docker images with docker images reveals that some required images are missing when compared to a healthy environment.

Non-working environment:

Working environment:

Prerequisites

• Terraform Enterprise with a Replicated installation.

Cause

The issue is likely caused by a network problem that prevents the Terraform Enterprise instance from pulling all required Docker images from the Replicated registry.

Solutions

Solution 1: Restore from Backup

If a recent backup of the instance is available, restore the changes to return to a stable environment.

Solution 2: Transfer Images from an Identical Environment

If an identical, working Terraform Enterprise environment exists, you can transfer the Docker images from the working environment to the non-working one.

1. Log in to the working environment, which must be on the same version of Terraform Enterprise. Save all Docker images to a tarball.

   $ docker save -o all-images.tar $(docker images --format "{{.Repository}}:{{.Tag}}")

2. Copy the all-images.tar file to the non-working environment.

3. On the non-working environment, load the images from the tarball.

   $ docker load -i all-images.tar

Solution 3: Manually Authenticate and Pull Images

If the other options are not available, investigate potential network traffic blocks with your network team. You can also perform the following steps to manually authenticate with the Replicated registry and pull the images.

1. You will need your Replicated license file (.rli). Unarchive it.

   $ tar zxf replicated_license.rli
   $ ls
   replicated_license.rli key.signature license.json license.signature signing_key.pub

2. Retrieve the license ID from the license.json file. You must have jq installed.

   $ cat license.json | jq .
   {
     "id": "xxxxbbbb5e17469e509e01aeaaaaxxxx",
     ## ...
   }

3. Log in to the Replicated registry using the license ID as both the username and password.

   $ docker login -u xxxxbbbb5e17469e509e01aeaaaaxxxx registry.replicated.com -p xxxxbbbb5e17469e509e01aeaaaaxxxx
   WARNING! Using --password via the CLI is insecure. Use --password-stdin.
   Login Succeeded

4. Test pulling a required image.

   $ docker pull registry.replicated.com/terraformenterprise/2rzxli7kinrk.hashicorp-ptfe-postgres

Outcome

After all required Docker images are successfully loaded or pulled, the Terraform Enterprise application should start and become fully operational.

Additional Information

• Terraform Enterprise (Replicated Installation) - Basic Troubleshooting Guide
• For more details on installation, refer to the official Terraform Enterprise documentation.