Problem
When Terraform encounters an error while writing to a state backend, it generates an errored.tfstate file and displays the following error in the HCP Terraform or Terraform Enterprise UI.
Error: Failed to persist state to backend The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file "errored.tfstate" in the current working directory. Running "terraform apply" again at this point will create a forked state, making it harder to recover. To retry writing this state, use the following command: terraform state push errored.tfstate
This errored.tfstate file allows you to restore the state by pushing this file as the latest state version. For runs executed by TFC Agents, which have replaced Terraform Build Workers, this guide outlines how to access the errored state file.
Prerequisites
- HCP Terraform or Terraform Enterprise version
v202309-1or newer. - HCP Terraform Agent version
1.12.1or newer (only relevant for agent execution mode).
Solutions
Solution 1: Recover State via API (Recommended)
During an apply run, if the HCP Terraform Agent detects that an errored.tfstate file was written, it uploads the file to the HCP Terraform or Terraform Enterprise platform.
To access this file, use the Recover a failed state upload after applying API endpoint. You must first obtain the external ID of the apply. Use the Runs API and extract the .data.relationships.apply.data.id attribute from the response.
- Retrieve the apply ID from the run data.
$ curl \ --header "Authorization: Bearer $TOKEN" \ https://app.terraform.io/api/v2/runs/<RUN_EXTERNAL_ID> | jq '.data.relationships.apply.data.id'
- Use the apply ID with the recovery endpoint to download the
errored.tfstatefile.
Solution 2: Manually Import Resources
If you are using versions of Terraform Enterprise or the HCP Terraform Agent lower than those specified in the prerequisites, the automatic upload capability is not available. In this case, to prevent a forked state, you must manually import the resources that were not saved to the state during the failed apply run.