Problem
When attempting to restore a Terraform Enterprise instance from a snapshot stored in an AWS S3 bucket, the operation fails with one of the following errors.
Error 1: Checksum Mismatch
Error restoring from snapshot: restore registry file "/var/lib/replicated/registry/sha256:06da004e6acdf55bea18c8dbfaa73c07ffffa1fc88c9ec845ccf0ac1c645cb24": checksum error: server:8c82acced149f7c455e98f4f0f9cf249b22819ea8ac578baafe342dcfdc334c4, local:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Error 2: Invalid Object State
snapshots/server/handlers.go:497 Error InvalidObjectState: The operation is not valid for the object's storage class status code: 403, request id: VZXZ4TANBQ2118SD, host id: UXE+khhbje423j4MPhEWsPjbUihGqE09GXHmazz+hgTTQFZAI87njwwcWFU83bWp+5ELqix5L40= reading file 066c9a08a28643f35ea716f7e0bd01c6
Cause
These errors typically occur when an S3 lifecycle policy has modified the objects that are part of the Terraform Enterprise snapshot.
Replicated snapshots are incremental and consist of multiple file objects stored within a specific path in an S3 bucket. For a restore operation to succeed, all of these files must be present and accessible in the Standard S3 storage class.
If an S3 lifecycle policy moves these files to a different storage class (such as Glacier) or deletes them, the snapshot becomes incomplete or inaccessible, causing the restore to fail.
Solutions
Solution 1: Restore Archived or Deleted Snapshot Files
Restore any snapshot files that were moved to a different storage class (e.g., Glacier) or deleted by an S3 lifecycle policy. After restoring the files to the Standard storage class, attempt the restore operation again.
Solution 2: Adjust S3 Lifecycle Policies
Review and remove any S3 lifecycle policies that archive or delete files within the S3 bucket and path used for Terraform Enterprise snapshots. Ensure that the snapshot data remains in the Standard storage class and is not subject to automated modification or deletion.
Additional Information
- For more details on Terraform Enterprise snapshot and restore procedures, please refer to the official documentation on this topic.
- To learn more about S3 storage classes and lifecycle policies, consult the relevant AWS S3 documentation.