Problem
A data retention policy is configured for Terraform Enterprise, but old state file versions and deletion markers persist in the configured AWS S3 storage bucket.
Cause
This issue occurs due to the interaction between Terraform Enterprise's deletion process and the bucket's S3 versioning feature. The root cause is a combination of factors:
- Terraform Enterprise Deletion Process: When a state file is deleted according to the retention policy, TFE marks it as "soft-deleted" for one week before transitioning it to a "permanently deleted" state.
- S3 Bucket Versioning: When versioning is enabled, deleting an object in S3 does not permanently remove it. Instead, S3 creates a "delete marker" that becomes the current version, effectively hiding previous versions of the object.
- Lifecycle Management: Terraform Enterprise does not manage S3 object versions or delete markers. Without a specific S3 lifecycle policy, these non-current versions and delete markers will persist indefinitely, even after TFE considers them "permanently deleted".
Solution
Because Terraform Enterprise does not manage the lifecycle of object versions or markers in the S3 bucket, you must implement an S3 lifecycle policy to manage them directly.
Configure a lifecycle policy on the S3 bucket that expires non-current object versions and removes delete markers. You can use the NoncurrentVersionExpiration action in your S3 lifecycle policy to automatically delete non-current object versions after a specified period, such as 30 days. According to AWS documentation, this action also handles the removal of expired object delete markers.
This ensures that state files considered permanently deleted by Terraform Enterprise are also physically removed from the S3 bucket over time.