Problem
In some scenarios, the bundled version of PostgreSQL used within the backup and restore container within Terraform Enterprise may be newer than the version of PostgreSQL you’re using on your external PostgreSQL DB, causing incompatibilities with the backup and restore API between hosts leading to errors such as 502 bad
gateway error
or pg_restore: error: could not read from input file: end of file
.
Prerequisites
- Known affected version of TFE v202207-2.
- External PostgreSQL on 11.
Cause
The incompatibilities are caused by the difference in versions between the bundled PostgreSQL and external PostgreSQL.
Solution
Notes:
Primary TFE host will be referred to as Source
The new TFE host will be referred to as Dest
.
During the docker tag stage, you will need to specify the IP address for the specific instance you're working on, otherwise there may be issues with TFE recognizing the image on that node.
On both instances, run docker image ls | grep backup
to confirm the image tag. On my instance, it was 291-282f635
which should be the same on your side.
The following steps will need to be completed on both the Source
and Dest
instances:
- Create a Dockerfile by running
touch Dockerfile
in the directory of your choice and add the following contents to the Dockerfile:
FROM registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635 as backup-restore
FROM alpine:3.10
COPY --from=backup-restore /usr/bin/backup-restore /usr/bin/backup-restore
COPY --from=backup-restore /usr/local/bin/redli /usr/local/bin/redli
COPY --from=backup-restore /etc/nsswitch.conf /etc/nsswitch.conf
RUN apk add --no-cache ca-certificates bash curl tar postgresql-client redis openssl
ENTRYPOINT ["backup-restore"]
- Build the Docker image with the following command:
docker build -t localhost/tfe-backup-restore:latest .
- Once the image has been built on both the
Source
andDest
instances, run the following command on theSource
instance, replace$REPLACE_WITH_SOURCE_IP
with the IP of the Source host IP:
docker tag localhost/tfe-backup-restore:latest $REPLACE_WITH_SOURCE_IP:9874/hashicorp-tfe-backup-restore:291-282f635
docker tag localhost/tfe-backup-restore:latest registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635
- Run the following command on the
Dest
instance, replacing$REPLACE_WITH_DEST_IP
with the IP of the Dest host IP:
docker tag localhost/tfe-backup-restore:latest $REPLACE_WITH_DEST_IP:9874/hashicorp-tfe-backup-restore:291-282f635
docker tag localhost/tfe-backup-restore:latest registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore:291-282f635
- On the
Source
instance, restart Terraform Enterprise using the following commands:
replicatedctl app stop
watch replicatedctl app status # Wait for the status to change to stopped
replicatedctl app start
- On the
Dest
instance, clear its database of any content (if necessary) and restart Terraform Enterprise. - On both instances, run
docker image ls | grep backup
. The output should look similar to the below example:
$ docker image ls | grep backup
10.0.164.150:9874/hashicorp-tfe-backup-restore 291-282f635 3716ae4d9f66 7 minutes ago 56.7MB
localhost/tfe-backup-restore latest 3716ae4d9f66 7 minutes ago 56.7MB
registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore 291-282f635 3716ae4d9f66 7 minutes ago 56.7MB
10.0.164.150:9874/hashicorp-tfe-backup-restore <none> 9ccb9d39a453 8 months ago 57.5MB
registry.replicated.com/terraformenterprise/jbc6stpmclxrk.hashicorp-tfe-backup-restore <none> 9ccb9d39a453 8 months ago 57.5MB
- On both instances, run
docker exec -it tfe-backup-restore pg_restore --version
- It should show version 11 if successful
$ docker exec -it tfe-backup-restore pg_restore --version
pg_restore (PostgreSQL) 11.12
- Once this point has been reached, you can retry your Backup and Restore API command from the beginning.
Outcome
By following the steps outlined in this article, you can resolve the incompatibility issue between the bundled PostgreSQL and external PostgreSQL, allowing you to successfully perform backups and restores between hosts.