Migrate TFE from Mounted Disk to External Services mode with Backup/Restore API

Introduction

Terraform Enterprise does not have a direct method to switch an existing installation from Mounted Disk mode to External Services mode. The only supported migration path is to use the backup and restore API to move application data to a new installation configured for External Services.

This guide provides the procedure to migrate Terraform Enterprise application data from a source instance to a new destination instance using this backup and restore process.

Prerequisites

• The destination Terraform Enterprise installation must be a new, running instance with no existing application data.
• The versions of Terraform Enterprise and its underlying PostgreSQL database must be identical on both the source and destination instances.
• After completing a restore, you must restart the Terraform Enterprise application before it can use the restored data.

Procedure

Step 1: Set environment variables for API tokens

The backup and restore API requires a specific authorization token, which is separate from user, team, or organization tokens. You must retrieve this token from the Replicated settings dashboard for both the source and destination Terraform Enterprise installations.

1. Access the Replicated console on each instance at https://<tfe-host>:8800.
2. Navigate to the Settings page and locate the Backup Token (backup_token).

3. Export the tokens as environment variables. Replace <source_token_value> and <destination_token_value> with your retrieved tokens.

   $ export SOURCE_TFE_TOKEN=<source_token_value>
   $ export DESTINATION_TFE_TOKEN=<destination_token_value>

Step 2: Create a backup payload file

The backup API encrypts the backup file. You must provide a payload containing a password during the backup, which is also required to decrypt the file during the restore.

Create a file named payload.json with one of the following options. Replace <password> with a secure password.

Option 1: Back up all data

This option includes all application data, including object storage.

{
  "password": "<password>"
}

Option 2: Skip object storage data

This option excludes object storage data from the backup, which can significantly reduce backup and restore times for large installations. You will need to manually migrate this data later.

{
  "password": "<password>",
  "skip_object_storage": true
}

Step 3: Create the backup

Send a POST request to the backup API on the source instance. This command uses the source token and the payload.json file to create an encrypted backup file named backup.blob.

Note: For best performance and to avoid network disconnects, run the backup and restore commands from a server located in the same network as your Terraform Enterprise installations.

Replace <source-tfe-host> with the fully qualified domain name of your source instance.

$ curl \
  --header "Authorization: Bearer $SOURCE_TFE_TOKEN" \
  --request POST \
  --data @payload.json \
  --output backup.blob \
  https://<source-tfe-host>/_backup/api/v1/backup

A successful request returns no errors and creates the backup.blob file in your current directory.

Step 4: Stop the source TFE application

To prevent any data changes after the backup is complete, stop the application on the source Terraform Enterprise instance.

Connect to the source instance via SSH and run the following command.

$ replicatedctl app stop

Step 5: Restore the backup

Send a POST request to the restore API on the destination instance. This command uploads the backup.blob file and the payload.json configuration to the new instance.

Replace <destination-tfe-host> with the fully qualified domain name of your new instance.

$ curl \
  --header "Authorization: Bearer $DESTINATION_TFE_TOKEN" \
  --request POST \
  --form config=@payload.json \
  --form snapshot=@backup.blob \
  https://<destination-tfe-host>/_backup/api/v1/restore

This process may take a significant amount of time depending on the size of your data.

Step 6: Manually upload object storage data (Conditional)

If you chose to skip object storage in the backup payload (Option 2), you must manually copy the data from the source instance's disk to your external object storage (e.g., Amazon S3).

1. Connect to the source TFE instance via SSH and find the disk path.

   $ replicatedctl app-config export | grep "disk_path" -A1

2. Configure your AWS credentials as environment variables.

   $ export AWS_ACCESS_KEY_ID=<your_access_key_id>
   $ export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>
   $ export REGION=<your_region>

3. Copy the data from the disk path to your S3 bucket. The following commands cover common directories.

   Note: The folder structure may vary based on your application version and usage. Ensure all contents under the <disk_path>/aux/archivist/ directory are copied to their corresponding prefixes in the destination bucket.

   ## Replace <disk_path> and <s3_bucket> with your values
   $ cd <disk_path>/aux/archivist/terraform
   $ aws s3 cp . s3://<s3_bucket>/archivistterraform --recursive
   
   $ cd <disk_path>/aux/archivist/sentinel
   $ aws s3 cp . s3://<s3_bucket>/archivistsentinel --recursive
   
   $ cd <disk_path>/aux/archivist/plan-export
   $ aws s3 cp . s3://<s3_bucket>/archivistplan-export --recursive
   
   $ cd <disk_path>/aux/archivist/policy-set-versions
   $ aws s3 cp . s3://<s3_bucket>/archivistpolicy-set-versions --recursive

Step 7: Restart the destination TFE application

After the restore is complete, you must restart the application on the new Terraform Enterprise instance for the changes to take effect.

1. Connect to the new TFE instance via SSH and stop the application.

   $ replicatedctl app stop

2. Monitor the status until the application is fully stopped.

   $ watch replicatedctl app status

3. Start the application.

   $ replicatedctl app start

Once the application is online, the new installation will mirror the source installation. If the hostname of your Terraform Enterprise instance has changed, you must reconfigure your VCS connections.

Additional Information

• For more details on the backup and restore process, refer to the official Terraform Enterprise Backup and Restore documentation.