[TFE] Restoring a state file from a versioned s3 bucket

Introduction

State files stored in an S3 bucket by Terraform Enterprise (TFE) are encrypted. When you delete a workspace, a key component of the encryption is also deleted, which prevents the state file from being decrypted directly.

This guide explains how to recover the state file of a deleted workspace if you have S3 bucket versioning enabled and a database backup from before the deletion. The process involves creating a temporary TFE environment to decrypt and retrieve the state file.

Prerequisites

Before you begin, ensure you have the following:

• S3 bucket versioning must be enabled on the bucket storing TFE state files.
• A database snapshot from a time before the workspace was deleted.
• The Encryption Password (enc_password) from the original TFE environment.

Procedure

Step 1: Recreate the Temporary TFE Environment

First, you will create a new, temporary TFE instance using a copy of your object storage and a restored database snapshot.

1. Copy the entire contents of your current TFE S3 bucket to a new S3 bucket. The following command provides an example of syncing two S3 buckets.

   $ aws s3 sync --quiet s3://current-bucket s3://new-bucket

2. Create a new database instance from a snapshot that was taken before the workspace was deleted.
3. Install a new TFE instance. The TFE version must match the version that was running when the database snapshot was taken. For installation instructions, refer to the Interactive Terraform Enterprise Installation or Automated Terraform Enterprise Installation documentation.
4. During the installation, provide the following configuration settings:
   • Ensure you use the Encryption Password (enc_password) from your original environment.
   • Point the Object Storage (s3_bucket) to the new S3 bucket you created.
   • Point the PostgreSQL Configuration (pg_netloc) to the new database instance you restored.

Once the new TFE environment is running, you should be able to see the deleted workspaces in the UI.

Step 2: Restore and Download the State File

Next, you will locate the deleted state file in the original S3 bucket, upload it to the new bucket, and download the decrypted version from the temporary TFE environment.

1. In the temporary TFE UI, navigate to the workspace you need to recover and select the States tab.
2. Click on the state version you want to retrieve.
3. Record the state ID, which has a format like sv-xxxxxxxxxxxxxxxx.
4. In the AWS Management Console, navigate to your original TFE S3 bucket.
5. Enable Show versions for the bucket objects.

6. Locate the state file using the state ID you recorded. The path will follow this structure:

   archivistterraform/states/<state-id>/<hash>/<encrypted-file-name>

7. Find the version of the encrypted state file that was deleted. Its version ID in S3 will appear as null. Download this file.
8. In your new S3 bucket, recreate the same directory structure (archivistterraform/states/<state-id>/<hash>/) and upload the state file you just downloaded.
9. Return to the temporary TFE UI, in the state version view where you found the state ID.
10. Click the Download button to download the decrypted state file.

After downloading the state file, review its contents to verify it is correct.

Step 3: Recreate the Original Workspace

Finally, you can recreate the workspace in your production TFE environment and upload the recovered state file.

1. In your production TFE environment, create a new workspace with the same name as the one that was deleted.
2. Migrate the recovered state file to the new workspace by following the procedure in Migrate Workspace State Using the Terraform Enterprise API.