Error uploading state: Conflict The MD5 hash of the state provided does not match what is currently known for the same serial number

Problem

A remote Terraform run or state operation in Terraform Enterprise or HashiCorp Cloud Platform Terraform fails with an error resembling one of the following:

Error uploading state: Conflict

The MD5 hash of the state provided does not match what is currently
known for the same serial number

The MD5 parameter passed does not match the MD5 of the provided state

The serial provided in the state file is not greater than the serial
currently known remotely. This means that there may be an update performed
since your version of that state. Update to the latest state to resolve
conflicts and try again

Solution

These issues can be resolved by pushing a new state with an incremented serial number to the Terraform Enterprise/HCP Terraform Workspace.

Steps for doing so with shell and Powershell commands against the TFE/HCP API can be found at the KB article How-to Create a State Version Using the API.

For MacOS users, modified instructions are included here:

1. Set an environment variable TOKEN to a valid Terraform Cloud API token with admin permissions to the workspace (i.e., user token belonging to your user).
2. Download the current state version from the workspace using our API, replacing the value of <WORKSPACE_ID> with the Workspace's ID:

   • curl \
     --header "Authorization: Bearer $TOKEN" \
     --header "Content-Type: application/vnd.api+json" \
     https://app.terraform.io/api/v2/workspaces/<WORKSPACE_ID>/current-state-version

   • The response will be a JSON object with an attribute hosted-state-download-url. Run the following command to download the state file from that URL.

     curl -o "current-state.tfstate" "<URL>"

3. Alternatively, you can download the desired state file directly from the Workspace "States" tab, but note the latest serial # in the available state files.
4. Increment the serial property of the state file by two. For example, if the state file shows 385, increment to 387. This avoids scenarios where an existing state version entry may already exist for the next serial number but not show in the UI.
5. Compute the MD5 sum of the file:
   

   • md5 current-state.tfstate

6. Base64 encode the file:
   

   • base64 -i current-state.tfstate

7. Create a payload.json file for the API request with the following content, replacing the value in "<>" with the values from the earlier steps:

   • {
     "data": {
     
       "type":"state-versions",
       "attributes": {
         "serial": <UPDATED_SERIAL_VALUE_IN_STATE_FILE>,
         "lineage": "<LINEAGE_VALUE_IN_STATE_FILE>",
         "md5": "<MD5SUM>",
         "state": "<BASE64_ENCODED_STATE>"
       }
      }
     }

8. Lock the workspace by navigating to "Settings" > "Locking".
9. Use the API to create a new state version:

   • curl \
     --header "Authorization: Bearer $TOKEN" \
     --header "Content-Type: application/vnd.api+json" \
     --request POST \
     --data @payload.json \
     https://app.terraform.io/api/v2/workspaces/<WORKSPACE_ID>/state-versions

Additional Information

These errors can also be resolved using Terraform's built in state pull/push commands, but using the API as above can avoid issues due to validation and Terraform version mismatches.