[TFE] Migrating S3 storage between providers - missing S3 object storage data

Introduction

Archivist internally supports a prefix configuration option based on the active S3, Azure, or Google store. The prefix is set differently according to the store type:

• Type S3 sets the prefix dynamically (either archivist or archivist/).

• Type Azure sets the prefix to archivist.

• Type Google sets the prefix to archivist/.

Problem

In some situations when moving Terraform Enterprise data between S3 data stores, the data may appear to be missing when attempting to start the instance. This can occur even if the data was successfully migrated to the new data store and the application configuration was updated with the new S3 endpoints.

Cause

This happens because the Terraform Enterprise application may have previously stored the S3 object data using a different archivist prefix depending on the S3 data store that was previously used.

Possible Related  Error Messages

• failed unpacking terraform config: failed to uncompress slug

Overview of possible solutions

With the archivist prefix, object storage data looks like this:

.
├── archivistsentinel
│   ├── output
│   └── policies
└── archivistterraform
    ├── json-plans
    ├── json-provider-schemas
    ├── logs
    ├── plans
    ├── slugs
    └── states

With the archivist/ prefix, object storage data looks like this:

.
└── archivist
    ├── sentinel
    │   ├── output
    │   └── policies
    └── terraform
        ├── json-plans
        ├── json-provider-schemas
        ├── logs
        ├── plans
        ├── slugs
        └── states

With the "" prefix, object storage data looks like this:

.
├── sentinel
│   ├── output
│   └── policies
└── terraform
    ├── json-plans
    ├── json-provider-schemas
    ├── logs
    ├── plans
    ├── slugs
    └── states

Solutions

There is a way to migrate object storage data from one prefix to the other.

Before proceeding, ensure that you have backups/snapshots of your object storage data in case reversion is necessary.

1. Data using the archivist prefix can be migrated to use the archivist/ prefix like so:

mkdir -p archivist/terraform
mkdir -p archivist/sentinel
mv archivistterraform/* archivist/terraform/
mv archivistsentinel/* archivist/sentinel/
rm -rf archivistsentinel archivistterraform

2. Similarly, data using the archivist/ prefix can be migrated to use the archivist prefix like so:

mkdir -p archivistterraform
mkdir -p archivistsentinel
mv archivist/terraform/* archivistterraform/
mv archivist/sentinel/* archivistsentinel/
rm -rf archivist

Outcome

Once the data in the object store has been migrated to the correct prefix for the applicable data store that you're using, restart the Terraform Enterprise application. At this point your data should be visible.

Additional Information

Please contact HashiCorp Support for more information.