Error: "flatmap states cannot be unmarshaled..."

Introduction

When migrating to a higher Terraform version you should follow the upgrade steps as documented in the upgrade guide found here.

Problem

When upgrading to a higher terraform version you can end up with the following error during terraform plan/apply:

flatmap states cannot be unmarshaled, only states written by Terraform 0.12
and higher can be unmarshalled

Cause

Best practice is to keep the provider versions locked when using a new major release of terraform. The new terraform version will change the state file format. A new provider version can also make changes to the state file format. If you combine both updates at once, you will hit the flatmap states cannot be unmarshaled error.

The following is an example of hitting this error:

• Configuration example

resource"random_password""password" {
  length=16
  special=true
  override_special="!#$%&*()-_=+[]{}<>:?"
}

output"password" {
  value="${random_password.password.id}"
}

• Using terraform version 0.11
• Initialize terraform

terraform init

NOTE: Notice that during initialization, Terraform downloaded the random provider version 2.3.1.

- Downloading plugin for provider "random" (2.3.1)...

• Apply the configuration

terraform apply

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

• Remove your .terraform directory, so you start without any previous providers

rm -rf .terraform

• Switch to terraform version 0.12.

terraform version

Terraform v0.12.31

• Initialize terraform

terraform init

NOTE: Notice that during initialization, Terraform downloaded the random provider version 3.4.3.

- Downloading plugin for provider "random" (hashicorp/random) 3.4.3...

• Apply the configuration

terraform apply

Error: Unable to Read Previously Saved State for UpgradeResourceState

There was an error reading the saved resource state using the prior resource
schema defined for version 0 upgrade.

Please report this to the provider developer:

flatmap states cannot be unmarshaled, only states written by Terraform 0.12
and higher can be unmarshaled

Solution

• Remove your .terraform directory to start again without any previous providers downloaded.

rm -rf .terraform

• Add the provider version and lock it to the release used by the previous terraform version.

provider"random" {
  version="2.3.1"
}

resource"random_password""password" {
  length=16
  special=true
  override_special="!#$%&*()-_=+[]{}<>:?"
}

output"password" {
  value="${random_password.password.id}"
}

• Switch to terraform version 0.12.

terraform version

Terraform v0.12.31

• Initialize terraform

terraform init

NOTE: Notice that during initialization, Terraform downloaded the random provider version 2.3.1.

- Downloading plugin for provider "random" (hashicorp/random) 2.3.1...

• Apply the configuration

terraform apply
random_password.password: Refreshing state... [id=none]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

• Change the provider block to use a higher provider version.

provider"random" {
  version="3.4.2"
}

• Initialize terraform with the - upgrade parameter. This will download the new provider version changed in your provider block.

terraform init -upgrade

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 3.4.2...

• Apply the configuration

terraform apply
random_password.password: Refreshing state... [id=none]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Additional Information

• If you have locked the provider versions and still experience errors, please contact HashiCorp Support.

• Terraform upgrade guides can be found here.