Introduction
You're attempting to perform a terraform destroy operation or are in the midst of replacing a resource, and run into an error like this:
Error: error updating Secrets Manager Secret
"arn:aws:secretsmanager:<region>:<acct-id>/<resource>"
...
InvalidRequestException: You can't perform this operation on the secret because it was
marked for deletion.
In your troubleshooting, you also could've checked the AWS Console for the secret and found it deleted, but still see the secret in your state file, and might be led to think that the issue is with Terraform.
Cause & Possible Solutions
This AWS API error is a known behavior per AWS design. AWS Secrets Manager Secrets have a default retention (recovery window) period of 30 days after deletion to give you the chance to recover it - in case it was deleted by mistake or you change your mind. So even if it shows on the AWS console as deleted, it still exists in AWS's API records, that's why Terraform has not removed it from state yet.
==> That recovery window is configurable on the AWS Console but with a minimum recovery window of 7 days, however AWS has an API call that allows you to delete a secret without recovery - called ForceDeleteWithoutRecovery [1], and to implement it using Terraform, you change the recovery_window_in_days parameter [2] of the aws_secretsmanager_secret resource to 0.
- Please note that you can only do this API call on an apply BEFORE a deletion (i.e terraform destroy), but if you've performed the destroy without knowing about this, you have 2 other options:
a. Perform the aws secretsmanager delete-secret cli command [3] to remove the retention window AND then perform the terraform refresh command [4] to reconcile your state file. So you won't see the error again, OR
b. Recover the secret on your AWS console, import it back into your config [5], set its recovery_window_in_days parameter to 0, and then do your terraform destroy
Any of these approaches should work for you, however please be mindful of the caveat below.
IMPORTANT NOTE: Additional Troubleshooting Scenarios and Solution
After deleting the secret (even after following the steps above), it is recommended that you wait about an hour or 2 for AWS to delete the secret permanently on their end before moving on, otherwise you might run into this error:
Error: error reading Secrets Manager Secret Version
("arn:aws:secretsmanager:<region>:<acct-id>/<resource>"):
DecryptionFailure: Secrets Manager can't decrypt the secret value:
arn:aws:kms:<region>:<acct_id>:key/<key_id> is pending deletion. (Service: AWSKMS;
Status Code: 400; Error Code: KMSInvalidStateException; Request ID: <request-id>;
Proxy: null)
==> This is because the secret deletion is an eventual delete on AWS' side so there's some delay in when that gets done. This is explained in the AWS documentation [6] excerpt below:
"Secrets Manager performs the actual deletion with an asynchronous background process, so there might be a short delay before the secret is permanently deleted. If you delete a secret and then immediately create a secret with the same name, use appropriate back off and retry logic."
==> If after waiting an hour or so, you're still running into issues, then try performing terraform state rm [7] to remove the secret from state, and import it back [8] into your config, OR consider contacting AWS or HashiCorp Support for further assistance.
Link References
[1] https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_DeleteSecret.html#:~:text=in%20JSON%20format.-,forcedeletewithoutrecovery,-Specifies%20whether%20to
[2] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret#recovery_window_in_days
[3] https://awscli.amazonaws.com/v2/documentation/api/latest/reference/secretsmanager/delete-secret.html#:~:text=day%20recovery%20window.-,--force-delete-without-recovery,-%7C%20--no-force
[4] https://www.terraform.io/cli/commands/refresh
[5] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret#import
[6] https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_DeleteSecret.html
[7] https://www.terraform.io/cli/commands/state/rm