Problem
Data retention has been configured for Terraform Enterprise. However, old state files with deletion markers are still present in storage.
Cause
A state file is set to be deleted based on the data retention policy established in TFE. It first goes into a "soft-deleted" state. If it's not restored within a week, it transitions to a "permanently deleted" state, at which point TFE intends for the files to be permanently removed.
Deletion Behavior with Versioning: The storage in question has versioning enabled, which alters the deletion process. With versioning:
-
- A "delete marker" is created for the object, which effectively hides it from view but does not delete the actual version of the object.
- If a lifecycle policy is not in place to handle these delete markers and previous versions, they will remain indefinitely despite being considered "permanently deleted" from TFE's perspective.
Solution:
Since TFE does not manage the deletion of object versions or markers in storage, a lifecycle policy should be implemented to manage these. This policy should specifically target the expiration of non-current object versions and the removal of deleted markers.
To ensure the actual deletion of non-current versions and associated delete markers, it is recommended that you use the `NoncurrentVersionExpiration` action in your lifecycle policy. This action can be configured to automatically delete non-current object versions after a specified period, e.g., 30 days after they become noncurrent. AWS documentation indicates that this process will also handle the automatic removal of expired delete markers.
Outcome
In conclusion, to achieve permanent deletion of old state files in an S3 bucket with versioning enabled, you'll need to implement an S3 lifecycle policy that specifically targets the cleanup of non-current versions and delete markers, as TFE's deletion process does not automatically manage these aspects in S3.