Introduction
When Terraform encounters an error writing to state, it generates an errored.tfstate file, displaying the following error in the Terraform Cloud/Enterprise UI.
Error: Failed to persist state to backend
The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file "errored.tfstate" in the current working directory.
Running "terraform apply" again at this point will create a forked state, making it harder to recover.
To retry writing this state, use the following command:
terraform state push errored.tfstate
This allows for the restoration of the state under typical conditions by pushing this file as the latest state version. Traditionally, when Terraform Cloud and Terraform Enterprise runs were executed using Terraform Build Workers, this file would be dumped in the apply log. This guide outlines how to access the errored state file for runs executed by TFC Agents, which have replaced Terraform Build Workers.
Prerequisites
- Terraform Cloud or Terraform Enterprise >= v202309-1
- Terraform Cloud Agent >= 1.12.1 (only relevant for agent execution mode)
Procedure
During an apply run, if the Terraform Cloud Agent detects an errored.tfstate file was written, it will upload the file to the Terraform Cloud/Enterprise platform. To access this file, use the Recover a failed state upload after applying API. To obtain the external ID of the apply, use the Runs API and extract the .data.relationships.apply.data.id attribute.
curl \
--header "Authorization: Bearer $TOKEN" \
https://app.terraform.io/api/v2/runs/<RUN_EXTERNAL_ID> | jq '.data.relationships.apply.data.id'
Workaround
Terraform Enterprise and Terraform Cloud Agent versions lower than those specified above do not have this capability. In this case, the path forward to prevent a forked state is to import the resources which were not saved to state during the failed apply run.