Migrating Terraform Workspace State Across Organizations and Backends (Terraform CLI, HCP Cloud, and Terraform Enterprise)

Avatar
Darrell F. Williams
  • Updated

Use Case

This article outlines CLI-based methods to migrate Terraform state across different organizations, backends, or even between Terraform Enterprise (TFE) instances using the TFE API. These workflows support common scenarios such as:

  • Restoring or modifying state within a Terraform Cloud or Enterprise workspace using terraform state pull and push, particularly when a revert or manual correction is needed (Objective 1).

  • Migrating a workspace to a different organization in Terraform Cloud or Enterprise, often required during organizational restructuring or project transitions (Objective 2).

  • Transferring state between separate Terraform Enterprise instances, enabling use cases like regional failover, environment duplication, or consolidating infrastructure under a new TFE instance (Objective 3).

Each objective below provides detailed CLI-driven steps tailored to these scenarios and applicable across Terraform CLI, Cloud, and Enterprise environments.

Objective 1 

The current versions of Terraform Enterprise (TFE) and HCP Terraform do not provide a feature to revert state within the application. You can, however, use Terraform on the command line to push a state file into a workspace to become the current state.

Procedure

  1. To obtain the desired state, navigate to the workspace’s States list, select the desired state, and click "Download" or “Download raw” to download the raw state file to your local machine.
  2. Configure Terraform locally with the cloud block and run terraform init as described in that article.
  3. With the cloud block configured and Terraform initialized, the state mv and state rm commands are supported. These commands enable direct modification of the state within the Terraform Enterprise or Terraform Cloud workspace.
  4. In some scenarios, such as if the state in Terraform Enterprise or Terraform Cloud was updated incorrectly or corrupted, direct state manipulation may be required. In these cases, you may pull the state file to the local directory using for editing using the following command.
    $ terraform state pull

          The state may then be manipulated and subsequently uploaded using the following command, where /path/to/statefile.json is the updated state file.

    $ terraform state push /path/to/statefile.json

Additional Information

  • Terraform’s state (mv|rm|push) and also Terraform’s import commands will modify the version of Terraform CLI in the state file to the version from which you are running the CLI command. Please make sure that you are using the same version of Terraform CLI as configured in the target workspace to avoid a conflict of the CLI version.
  • This is terraform state push and not terraform push. terraform push is deprecated and does not operate in the current version of TFE.
  • Please visit Import Terraform configuration tutorial for more sample of importing resources into Terraform state.
  • Please visit Manage Resources in Terraform State tutorial for more sample of managing resources in Terraform state.

Objective 2

If you need to migrate a workspace to a different organization.

Procedure

Please note you’ll need a user API token that has sufficient permissions in both organizations. (https://app.terraform.io/app/<your organization>/settings/authentication-tokens) for cloud or (https://<tfe-url>/app/<your organization>/settings/authentication-tokens)

Ensure that all of the settings (VCS, variables, SSH keys, Terraform version, etc) are the same in both the Original and Destination Organizations workspaces.

NOTE: The hostname attribute should be set to either Terraform Cloud such as app.terraform.io or Terraform Enterprise <tfe-hostname>.

  1. Download the same version of terraform used in the Organizations/workspaces locally
  2. Locally, create a remote.tf file of the origin Organization * Do not commit this to your VCS repository.*
terraform {
  cloud {    
    hostname = "<TFE Hostname>"
    organization = "<Originating Organization name>"

    workspaces {
      name = "<Originating workspace name>"
    }
  }
}
  1. From a command prompt where the remote.tf exists, log in into Terraform API from the command line by issuing terraform login
  2. Follow the instructions of the prompts, the second one will require the user API token
  3. Run the command terraform init to begin establishing communications between the Organizations
  4. Ensure the Destination Organization and Workspace exists and if not create them prior to continuing to step 7
  5. Modify the remote.tf to point to the Destination Organization:
terraform {
  cloud {    
    hostname = "<TFE Hostname>"
    organization = "<Destination Organization name>"

    workspaces {
      name = "<Destination workspace name>"
    }
  }
}
  1. Execute $terraform init -migrate-state to trigger the communication with the Destination Organization. Terraform should prompt to copy the state file to the new Organization workspace. Enter yes.
  2. Check the Destination Organization workspace for the new state file
  3. If there is no prompt to move the state file rename the state file to statefile.json
  4. Run terraform state push /path/to/statefile.json
  5. Check the Destination Organization workspace for the new state file

Objective 3

This objective provides step-by-step instructions on how to migrate a Terraform state file from one Terraform Enterprise (TFE) instance to another using the TFE API. By following these steps, you can seamlessly transfer your workspace state and continue managing your infrastructure on a different TFE instance.

Prerequisites

Before proceeding with the migration, ensure you have the following variables and information ready:

  • ${HOSTNAME_OLD}: The hostname for the source Terraform Enterprise instance.
  • ${HOSTNAME_NEW}: The hostname for the target Terraform Enterprise instance.
  • ${TOKEN_OLD}: The user API token for the source Terraform Enterprise instance.
  • ${TOKEN_NEW}: The user API token for the target Terraform Enterprise instance.
  • ${WORKSPACE_ID_OLD}: The workspace ID for the source workspace.
  • ${WORKSPACE_ID_NEW}: The workspace ID for the target workspace.

Procedure

Step 1: Gather Information from the Source Instance

  1. Obtain the hostname of the source Terraform Enterprise instance (referred to as ${HOSTNAME_OLD}) to interact with it via the API.
  2. Create a user API token for the source Terraform Enterprise instance (referred to as ${TOKEN_OLD}) to authenticate API interactions.
  3. Retrieve the workspace ID (referred to as ${WORKSPACE_ID_OLD}) of the workspace you want to migrate from the source instance. You may navigate to Settings > General in the TFE web interface to find the workspace ID.

Step 2: Download the Source State File

  1. Use the following curl command to retrieve the download URL for the state file of the source workspace:
curl --header "Authorization: Bearer ${TOKEN_OLD}" --header "Content-Type: application/vnd.api+json" https://${HOSTNAME_OLD}/api/v2/workspaces/${WORKSPACE_ID_OLD}/current-state-version
  1. Download the state file from the hosted-state-download-url obtained in the previous step. Save the file as ${WORKSPACE_ID_OLD}.tfstate in your current working directory.

Step 3: Gather Information from the Target Instance

  1. Obtain the hostname of the target Terraform Enterprise instance (referred to as ${HOSTNAME_NEW}) to interact with it via the API.
  2. Create a user API token for the target Terraform Enterprise instance (referred to as ${TOKEN_NEW}) to authenticate API interactions.
  3. Create a new workspace within the target Terraform Enterprise instance, ensuring it uses the same VCS provider, variables, and Terraform version as the source workspace.
  4. Lock the new workspace by navigating to Settings > Locking and clicking on the "Lock" button.
  5. Retrieve the workspace ID (referred to as ${WORKSPACE_ID_NEW}) for the newly created workspace in the target instance. Navigate to Settings > General in the TFE web interface to find the workspace ID.

Step 4: Prepare the Source State File for Migration

  1. Open the previously downloaded ${WORKSPACE_ID_OLD}.tfstate file and take note of the "serial" and "lineage" values.
  2. Compute the MD5 checksum of the state file using the following command:
md5sum ${WORKSPACE_ID_OLD}.tfstate
  1. Compute the base64 content of the state file using the following command or similar:
base64 -i ${WORKSPACE_ID_OLD}.tfstate | tr -d '\n'
  1. Create a file named ${WORKSPACE_ID_OLD}-payload.json with the following content, replacing the values in < > with your previously noted values. Ensure the "serial" attribute is not enclosed in double quotes, as it is a number.
{
  "data": {
    "type": "state-versions",
    "attributes": {
      "serial": <SERIAL>,
      "md5": "<MD5_CHECKSUM>",
      "lineage": "<LINEAGE>",
      "state": "<BASE64_CONTENT>"
    }
  }
}

Step 5: Upload the State File to the Target Workspace

Use the following curl command to upload the state file to the new Terraform Enterprise workspace in the target instance:

curl --header "Authorization: Bearer ${TOKEN_NEW}" --header "Content-Type: application/vnd.api+json" --request POST --data @${WORKSPACE_ID_OLD}-payload.json https://${HOSTNAME_NEW}/api/v2/workspaces/${WORKSPACE_ID_NEW}/state-versions

Step 6: Verify the Migration

Check if the new state file is listed in the target Terraform Enterprise workspace.

Step 7: Unlock the Target Workspace

Unlock the new workspace by navigating to Settings > Locking and clicking on the "Unlock" button.

Step 8: Repeat for Additional Workspaces

Repeat the entire process for each workspace state you wish to migrate to the target Terraform Enterprise instance.

Additional Information

For more information or troubleshooting, refer to the following resources:

Remember to adapt the steps to your specific environment and configurations. With these instructions, you can easily transfer your Terraform workspace states between different Terraform Enterprise instances via the API.

 

Articles in this section

See more

Related articles