Problem
Restoring a Terraform Enterprise instance from an S3 snapshot fails with one of the following errors:
Error restoring from snapshot: restore registry file "/var/lib/replicated/registry/sha256:06da004e6acdf55bea18c8dbfaa73c07ffffa1fc88c9ec845ccf0ac1c645cb24": checksum error: server:8c82acced149f7c455e98f4f0f9cf249b22819ea8ac578baafe342dcfdc334c4, local:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
snapshots/server/handlers.go:497 Error InvalidObjectState: The operation is not valid for the object's storage class
status code: 403, request id: VZXZ4TANBQ2118SD, host id: UXE+khhbje423j4MPhEWsPjbUihGqE09GXHmazz+hgTTQFZAI87njwwcWFU83bWp+5ELqix5L40= reading file 066c9a08a28643f35ea716f7e0bd01c6
Cause
- These errors occur if the s3 bucket where the Replicated snapshot is stored has been modified by an S3 policy.
-
-
Replicated snapshots are incremental and consist of multiple files in a s3 folder
-
All files in the folder must remain there in storage class "Standard"
-
If there is a policy on the folder that is moving files to Glacier or deleting them, the snapshots will fail to restore, as all the files are required for a complete restore.
-
Overview of possible solutions
Solutions:
-
Restore any files that were removed by a bucket policy and attempt the restore again.
-
Remove any policies that will modify/remove files in the S3 destination path for the snapshot.