How to manually rollback to an older version of the Backup and Restore API container due to compatibility issues – HashiCorp Help Center

Problem

When using the Terraform Enterprise (TFE) backup and restore API, operations may fail with errors such as 502 bad gateway error or pg_restore: error: could not read from input file: end of file. This can occur when the version of PostgreSQL bundled in the TFE backup and restore container is newer than the version used by an external PostgreSQL database, causing compatibility issues.

Prerequisites

Terraform Enterprise v202207-2 or a similarly affected version.
An external PostgreSQL database running version 11.

Cause

The incompatibility is caused by a version mismatch between the pg_restore utility in the TFE backup and restore container and the external PostgreSQL server.

Solution

This procedure manually builds a new backup and restore container image that uses an older, compatible version of PostgreSQL client tools. You must perform these steps on both the source and destination TFE instances.

Note: In this guide, the primary TFE host is referred to as the source instance, and the new TFE host is the destination instance.

Identify the current backup image tag.
On both TFE instances, run the following command to find the image tag. The image tag for the affected version is typically 291-282f635.
```
$ docker image ls | grep backup
```

Create a custom Dockerfile.

On both instances, create a file named Dockerfile in a working directory.

$ touch Dockerfile

Add the following content to the Dockerfile. This configuration uses a multi-stage build to copy the necessary binaries into an older alpine:3.10 base image that includes a compatible PostgreSQL client.

FROM registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635 as backup-restore

FROM alpine:3.10

COPY --from=backup-restore /usr/bin/backup-restore /usr/bin/backup-restore
COPY --from=backup-restore /usr/local/bin/redli /usr/local/bin/redli
COPY --from=backup-restore /etc/nsswitch.conf /etc/nsswitch.conf

RUN apk add --no-cache ca-certificates bash curl tar postgresql-client redis openssl

ENTRYPOINT ["backup-restore"]

Build the new Docker image.
On both instances, build the image from the Dockerfile.
```
$ docker build -t localhost/tfe-backup-restore:latest .
```

Tag the image for the source instance.

On the source instance, tag the newly built image to match TFE's expected image naming convention. Replace $REPLACE_WITH_SOURCE_IP with the actual private IP address of the source host.

$ docker tag localhost/tfe-backup-restore:latest $REPLACE_WITH_SOURCE_IP:9874/hashicorp-tfe-backup-restore:291-282f635
$ docker tag localhost/tfe-backup-restore:latest registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635

Tag the image for the destination instance.

On the destination instance, perform the same tagging step. Replace $REPLACE_WITH_DEST_IP with the actual private IP address of the destination host.

$ docker tag localhost/tfe-backup-restore:latest $REPLACE_WITH_DEST_IP:9874/hashicorp-tfe-backup-restore:291-282f635
$ docker tag localhost/tfe-backup-restore:latest registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635

Restart Terraform Enterprise.
On the source instance, restart the TFE application to apply the image change.
```
$ replicatedctl app stop
$ watch replicatedctl app status ## Wait for the status to change to stopped
$ replicatedctl app start
```
On the destination instance, clear its database of any content from previous failed attempts if necessary, and then restart the TFE application.

Verify the new image is in use.

On both instances, list the Docker images again. The output should show that the newly tagged images are available.

$ docker image ls | grep backup
## Output should be similar to:
10.0.164.150:9874/hashicorp-tfe-backup-restore                                         291-282f635   3716ae4d9f66   7 minutes ago    56.7MB
localhost/tfe-backup-restore                                                          latest        3716ae4d9f66   7 minutes ago    56.7MB
registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore   291-282f635   3716ae4d9f66   7 minutes ago    56.7MB
10.0.164.150:9874/hashicorp-tfe-backup-restore                                         <none>        9ccb9d39a453   8 months ago     57.5MB
registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore   <none>        9ccb9d39a453   8 months ago     57.5MB

Confirm the pg_restore version.
On both instances, execute a command inside the running tfe-backup-restore container to check the pg_restore version. A successful rollback will show version 11.
```
$ docker exec -it tfe-backup-restore pg_restore --version
pg_restore (PostgreSQL) 11.12
```

Outcome

After confirming the pg_restore version is 11.12 on both instances, the incompatibility issue is resolved. You can now retry the Terraform Enterprise Backup and Restore API commands.

Additional Information

For more details on the backup and restore process, please refer to the official Terraform Enterprise documentation on this topic.