[TFE] Migrating S3 storage between providers - missing S3 object storage data

Problem

When migrating Terraform Enterprise data between S3-compatible object storage providers, data may appear to be missing after starting the instance. This can occur even if the data was successfully copied to the new data store and the application configuration was updated with the new endpoints.

A related error message may include the following.

failed unpacking terraform config: failed to uncompress slug

Prerequisites

Before proceeding, ensure that you have a complete backup or snapshot of your object storage data in case a rollback is necessary.

Cause

The Terraform Enterprise application uses an internal component called archivist to manage object storage. Depending on the object store type (S3, Azure, or Google), archivist uses a different directory prefix to store data. When migrating between providers, a mismatch between the old prefix structure and the one expected by the new configuration can cause Terraform Enterprise to be unable to locate its data.

The prefix is set differently according to the store type:

• S3: Sets the prefix dynamically to either archivist or archivist/.
• Azure: Sets the prefix to archivist.
• Google: Sets the prefix to archivist/.

Prefix Directory Examples

An archivist prefix results in the following directory structure.

.
├── archivistsentinel
│   ├── output
│   └── policies
└── archivistterraform
    ├── json-plans
    ├── logs
    ├── plans
    ├── slugs
    └── states

An archivist/ prefix results in the following directory structure.

.
└── archivist
    ├── sentinel
    │   ├── output
    │   └── policies
    └── terraform
        ├── json-plans
        ├── logs
        ├── plans
        ├── slugs
        └── states

Solutions

To resolve this issue, you must restructure the data in the object storage bucket to match the prefix format expected by the current Terraform Enterprise configuration.

Solution 1: Migrate from archivist to archivist/ prefix

If your data is in the archivistterraform and archivistsentinel format but needs to be in the nested archivist/ format, run the following commands in your object storage bucket.

$ mkdir -p archivist/terraform
$ mkdir -p archivist/sentinel
$ mv archivistterraform/* archivist/terraform/
$ mv archivistsentinel/* archivist/sentinel/
$ rm -rf archivistsentinel archivistterraform

Solution 2: Migrate from archivist/ to archivist prefix

If your data is in the nested archivist/ format but needs to be in the archivistterraform and archivistsentinel format, run the following commands in your object storage bucket.

$ mkdir -p archivistterraform
$ mkdir -p archivistsentinel
$ mv archivist/terraform/* archivistterraform/
$ mv archivist/sentinel/* archivistsentinel/
$ rm -rf archivist

Outcome

Once you have migrated the data in the object store to the correct prefix structure, restart the Terraform Enterprise application. The application should now be able to locate and display the migrated data.