Error: "flatmap states cannot be unmarshaled..."

Problem

When upgrading to a newer major version of Terraform, a terraform plan or terraform apply command may fail with the following error:

Error: Unable to Read Previously Saved State for Upgrade

ResourceState

There was an error reading the saved resource state using the prior resource
schema defined for version 0 upgrade.

Please report this to the provider developer:

flatmap states cannot be unmarshaled, only states written by Terraform 0.12
and higher can be unmarshaled

Cause

This error typically occurs when both the Terraform version and a provider version are upgraded in the same step. A major Terraform version upgrade (e.g., from 0.11 to 0.12) often changes the state file format. Concurrently, a provider upgrade can also introduce its own changes to how it stores information in the state file. Attempting both upgrades simultaneously can create a state format conflict that the new Terraform version cannot resolve, leading to the flatmap states cannot be unmarshaled error.

Solution

The recommended solution is to perform the upgrades in two separate stages: first, upgrade the Terraform version while keeping the provider version locked, and second, upgrade the provider version.

Procedure

1. Lock the provider version: In your configuration, add a provider block that explicitly locks the provider to the version that was used with your previous Terraform version. In this example, the random provider version 2.3.1 was used with Terraform 0.11.

   terraform {
     required_providers {
       random = {
         source  = "hashicorp/random"
         version = "2.3.1"
       }
     }
   }
   
   resource "random_password" "password" {
     length  = 16
     special = true
     override_special = "!#$%&*()-_=+[]{}<>:?"
   }
   
   output "password" {
     value = "${random_password.password.id}"
   }

2. Remove the local provider cache: Delete the .terraform directory to ensure a clean initialization.

   $ rm -rf .terraform

3. Initialize with the new Terraform version: Run terraform init using your upgraded Terraform binary. This will download the locked provider version (2.3.1 in this case).

   $ terraform init

4. Apply the configuration: Run terraform apply to allow the new Terraform version to update the state file correctly with the old provider schema.

   $ terraform apply
   
   Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

5. Update the provider version constraint: Modify the provider block to allow for a newer version.

   terraform {
     required_providers {
       random = {
         source  = "hashicorp/random"
         version = "~> 3.4"
       }
     }
   }

6. Upgrade the provider: Run terraform init -upgrade to download the latest provider version that matches your new constraint.

   $ terraform init -upgrade

7. Apply the final configuration: Run terraform apply again. The state is now fully upgraded to both the new Terraform version and the new provider version.

   $ terraform apply
   
   Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Additional Information

• For detailed instructions on upgrading Terraform, refer to the official Terraform upgrade guides.