Terraform Enterprise Replicated app won't start after upgrade with and error manifest unknown

Problem

After upgrading a Terraform Enterprise (TFE) instance installed via the Replicated method, the application does not start. The system logs display an error indicating that a Docker manifest is not found.

Error response from daemon: manifest for replicated/replicated:current not found: manifest unknown: manifest unknown.

A more detailed log from systemd shows the replicated.service failing and restarting repeatedly.

Dec 30 23:28:33 tfehost systemd[1]: Starting Replicated Service...
Dec 30 23:28:33 tfehost docker[21167]: Error: No such container: replicated
Dec 30 23:28:33 tfehost systemd[1]: Started Replicated Service.
Dec 30 23:28:33 tfehost docker[21185]: Unable to find image 'replicated/replicated:current' locally
Dec 30 23:28:34 tfehost docker[21185]: docker: Error response from daemon: manifest for replicated/replicated:current not found: manifest unknown: manifest unknown.
Dec 30 23:28:34 tfehost docker[21185]: See 'docker run --help'.
Dec 30 23:28:34 tfehost systemd[1]: replicated.service: main process exited, code=exited, status=125/n/a
Dec 30 23:28:34 tfehost systemd[1]: Unit replicated.service entered failed state.
Dec 30 23:28:34 tfehost systemd[1]: replicated.service failed.
Dec 30 23:28:41 tfehost systemd[1]: replicated.service holdoff time over, scheduling restart.
Dec 30 23:28:41 tfehost systemd[1]: Stopped Replicated Service.

Cause

This error occurs when the replicated.service systemd unit file has been manually edited to pin the Replicated container image to a specific version tag (e.g., stable-2.53.2) instead of the expected current tag.

During a TFE or Docker upgrade, the system attempts to pull the image tagged as replicated/replicated:current. If the service file specifies a different, hard-coded version, Docker cannot find the current manifest, leading to a startup failure.

Below is an example of an incorrect, manually edited service file compared to the expected configuration.

Expected Configuration:

## /etc/systemd/system/replicated.service
## ... some output omitted for brevity ...
$REPLICATED_OPTS \
replicated/replicated:current
ExecStop=/usr/bin/docker stop replicated
Restart=on-failure
RestartSec=7
## ...

Incorrect Configuration:

## /etc/systemd/system/replicated.service
## ... some output omitted for brevity ...
$REPLICATED_OPTS \
replicated/replicated:stable-2.53.2
ExecStop=/usr/bin/docker stop replicated
Restart=on-failure
RestartSec=7
## ...

Solutions

Solution 1: Avoid Manual Service File Edits

As a best practice, do not manually edit the TFE systemd service files. These files are managed by the TFE installation and upgrade processes. Manual modifications can cause unexpected failures during routine operations like upgrades.

Solution 2: Revert Manual Changes

If the service file was modified, revert it to its default state.

1. Open the service file for editing. The file is typically located at /etc/systemd/system/replicated.service.

   # vi /etc/systemd/system/replicated.service

2. Locate the line specifying the Docker image and ensure it points to the replicated/replicated:current tag.

3. After saving the changes, reload the systemd daemon to apply the new configuration.

   # systemctl daemon-reload

4. Restart the Replicated service.

   # systemctl restart replicated

Outcome

After reverting the service file and restarting the service, Terraform Enterprise will start successfully. You can then access the application and perform Terraform runs.

Additional Information

For more details on system requirements and upgrade procedures, please refer to the official Terraform Enterprise documentation.