How to manually rollback to an older version of the Backup and Restore API container due to compatibility issues

Problem

In some scenarios, the bundled version of PostgreSQL used within the backup and restore container within Terraform Enterprise may be newer than the version of PostgreSQL you’re using on your external PostgreSQL DB, causing incompatibilities with the backup and restore API between hosts leading to errors such as 502 bad gateway error or pg_restore: error: could not read from input file: end of file.

Prerequisites

• Known affected version of TFE v202207-2.
• External PostgreSQL on 11.

Cause

The incompatibilities are caused by the difference in versions between the bundled PostgreSQL and external PostgreSQL.

Solution

Notes:

Primary TFE host will be referred to as Source

The new TFE host will be referred to as Dest. 

During the docker tag stage, you will need to specify the IP address for the specific instance you're working on, otherwise there may be issues with TFE recognizing the image on that node.

On both instances, run docker image ls | grep backup to confirm the image tag. On my instance, it was 291-282f635 which should be the same on your side.

The following steps will need to be completed on both the Source and Dest instances:

• Create a Dockerfile by running touch Dockerfile in the directory of your choice and add the following contents to the Dockerfile:

FROM registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635 as backup-restore

FROM alpine:3.10

COPY --from=backup-restore /usr/bin/backup-restore /usr/bin/backup-restore
COPY --from=backup-restore /usr/local/bin/redli /usr/local/bin/redli
COPY --from=backup-restore /etc/nsswitch.conf /etc/nsswitch.conf

RUN apk add --no-cache ca-certificates bash curl tar postgresql-client redis openssl

ENTRYPOINT ["backup-restore"]

• Build the Docker image with the following command:

docker build -t localhost/tfe-backup-restore:latest .

• Once the image has been built on both the Source and Dest instances, run the following command on the Source instance, replace $REPLACE_WITH_SOURCE_IP with the IP of the Source host IP:

docker tag localhost/tfe-backup-restore:latest $REPLACE_WITH_SOURCE_IP:9874/hashicorp-tfe-backup-restore:291-282f635
docker tag localhost/tfe-backup-restore:latest registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635

• Run the following command on the Dest instance, replacing $REPLACE_WITH_DEST_IP with the IP of the Dest host IP:

docker tag localhost/tfe-backup-restore:latest $REPLACE_WITH_DEST_IP:9874/hashicorp-tfe-backup-restore:291-282f635
docker tag localhost/tfe-backup-restore:latest registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635

• On theSource instance, restart Terraform Enterprise using the following commands:

replicatedctl app stop
watch replicatedctl app status # Wait for the status to change to stopped
replicatedctl app start

• On theDest instance, clear its database of any content (if necessary) and restart Terraform Enterprise.
• On both instances, run docker image ls | grep backup. The output should look similar to the below example:

$ docker image ls | grep backup
10.0.164.150:9874/hashicorp-tfe-backup-restore 291-282f635 3716ae4d9f66 7 minutes ago 56.7MB
localhost/tfe-backup-restore latest 3716ae4d9f66 7 minutes ago 56.7MB
registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore 291-282f635 3716ae4d9f66 7 minutes ago 56.7MB
10.0.164.150:9874/hashicorp-tfe-backup-restore <none> 9ccb9d39a453 8 months ago 57.5MB
registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore <none> 9ccb9d39a453 8 months ago 57.5MB

• On both instances, run docker exec -it tfe-backup-restore pg_restore --version
  • It should show version 11 if successful

$ docker exec -it tfe-backup-restore pg_restore --version
pg_restore (PostgreSQL) 11.12

• Once this point has been reached, you can retry your Backup and Restore API command from the beginning.

Outcome

By following the steps outlined in this article, you can resolve the incompatibility issue between the bundled PostgreSQL and external PostgreSQL, allowing you to successfully perform backups and restores between hosts.