Terraform Enterprise restore from S3 snapshot fails with "checksum error:" or "InvalidObjectState"

Problem

Restoring a Terraform Enterprise instance from an S3 snapshot fails with one of the following errors:

Error restoring from snapshot: restore registry file "/var/lib/replicated/registry/sha256:06da004e6acdf55bea18c8dbfaa73c07ffffa1fc88c9ec845ccf0ac1c645cb24": checksum error: server:8c82acced149f7c455e98f4f0f9cf249b22819ea8ac578baafe342dcfdc334c4, local:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

snapshots/server/handlers.go:497 Error InvalidObjectState: The operation is not valid for the object's storage class
status code: 403, request id: VZXZ4TANBQ2118SD, host id: UXE+khhbje423j4MPhEWsPjbUihGqE09GXHmazz+hgTTQFZAI87njwwcWFU83bWp+5ELqix5L40= reading file 066c9a08a28643f35ea716f7e0bd01c6

Cause

• These errors occur if the s3 bucket where the Replicated snapshot is stored has been modified by an S3 policy.

• • Replicated snapshots are incremental and consist of multiple files in a s3 folder

  • All files in the folder must remain there in storage class "Standard"

  • If there is a policy on the folder that is moving files to Glacier or deleting them, the snapshots will fail to restore, as all the files are required for a complete restore.

Overview of possible solutions

Solutions:

• Restore any files that were removed by a bucket policy and attempt the restore again.

• Remove any policies that will modify/remove files in the S3 destination path for the snapshot.