Problem
A remote Terraform run or state operation in Terraform Enterprise or HCP Terraform fails with an error resembling one of the following messages.
Error uploading state: Conflict The MD5 hash of the state provided does not match what is currently known for the same serial number
The MD5 parameter passed does not match the MD5 of the provided state
The serial provided in the state file is not greater than the serial currently known remotely. This means that there may be an update performed since your version of that state. Update to the latest state to resolve conflicts and try again
Cause
These errors occur when the Terraform state file you are trying to upload has a serial number that is less than or equal to the serial number of the state already stored in the HCP Terraform or Terraform Enterprise workspace. This conflict prevents Terraform from overwriting a newer state with an older one, which protects against data loss.
Solutions
There are two primary methods to resolve this issue. The recommended approach is to use the API to manually create a new state version, as this bypasses certain client-side validations that can cause issues.
Solution 1: Manually Create a New State Version via the API
This procedure involves downloading the current state, manually incrementing its serial number, and uploading it through the API. The following steps use shell commands common on Linux and macOS.
For an alternative using PowerShell, refer to the KB article How-to Create a State Version Using the API.
-
Set an environment variable
TOKENto a valid HCP Terraform API token with admin permissions for the workspace.$ export TOKEN="<YOUR_API_TOKEN>"
-
Download the current state version from the workspace using the API. Replace
<WORKSPACE_ID>with your workspace's ID.$ curl \ --header "Authorization: Bearer $TOKEN" \ --header "Content-Type: application/vnd.api+json" \ https://app.terraform.io/api/v2/workspaces/<WORKSPACE_ID>/current-state-version
-
The response is a JSON object containing a
hosted-state-download-urlattribute. Use this URL to download the state file.$ curl -o "current-state.tfstate" "<PASTE_THE_URL_HERE>"
Alternatively, you can download the state file from the workspace's States tab in the UI. Note the latest serial number from the available state files.
- Open the downloaded
current-state.tfstatefile in a text editor and increment theserialproperty by at least two. For example, if the serial is385, change it to387. Incrementing by two helps avoid conflicts where an intermediate state version might exist but is not visible in the UI. -
Compute the MD5 hash of the modified state file.
## On macOS: $ md5 -q current-state.tfstate ## On Linux: $ md5sum current-state.tfstate | awk '{ print $1 }' -
Base64 encode the modified state file.
$ base64 < current-state.tfstate
-
Create a file named
payload.jsonwith the following content. Replace the bracketed values (<>) with theserialandlineagevalues from your modified state file, the MD5 hash, and the Base64 encoded string from the previous steps.{ "data": { "type": "state-versions", "attributes": { "serial": <UPDATED_SERIAL_VALUE_IN_STATE_FILE>, "lineage": "<LINEAGE_VALUE_IN_STATE_FILE>", "md5": "<MD5_HASH>", "state": "<BASE64_ENCODED_STATE>" } } } - Lock the workspace by navigating to Settings > Locking in the UI.
-
Use the API to create a new state version by posting the
payload.jsonfile.$ curl \ --header "Authorization: Bearer $TOKEN" \ --header "Content-Type: application/vnd.api+json" \ --request POST \ --data @payload.json \ https://app.terraform.io/api/v2/workspaces/<WORKSPACE_ID>/state-versions
After a successful API call, you can unlock the workspace and resume normal operations.
Solution 2: Use the Terraform CLI State Commands
You can also use the terraform state pull and terraform state push commands to resolve this issue.
- Ensure your local Terraform configuration is configured to use the correct remote backend for the workspace.
-
Pull the latest state from the remote workspace.
$ terraform state pull > current-state.tfstate
- Modify the
current-state.tfstatefile as needed. If you are trying to restore a previous state, replace the contents with your backup state file. Ensure theserialnumber is greater than the one currently stored in the workspace. -
Push the modified state file back to the workspace.
$ terraform state push current-state.tfstate
Additional Information
While using the Terraform CLI pull and push commands is a valid approach, using the API as described in Solution 1 can help avoid potential issues related to client-side validation or Terraform version mismatches between your local environment and the remote execution environment.