How to Migrate Terraform State into Terraform Cloud via CI/CD pipeline

Introduction

The Terraform state file is a critical component of Terraform operations, as it maintains the mapping between Terraform-managed resources and the real-world infrastructure they represent.

In some scenarios, teams use CI/CD pipelines to manage a large number of Terraform workspaces while storing their state files in a remote backend. Over time, a team may decide to adopt Terraform Cloud as their state backend and execution platform, creating the need to migrate existing Terraform state files into Terraform Cloud.

Expected Outcome

• The existing Terraform state is successfully migrated to the target Terraform Cloud workspace.

• New Terraform runs are visible in the Terraform Cloud workspace, and all subsequent changes are applied to the state stored in Terraform Cloud.

• The state stored in the previous backend is no longer updated by Terraform runs.

Prerequisites 

• An existing Terraform state stored in remote object storage.

• A CI/CD pipeline authenticated to:

  • The source backend - for reading the existing Terraform state.

  • Terraform Cloud - for writing the state to the destination workspace.

Use Case

This article focuses on migrating Terraform state from Google Cloud Storage (GCS) to Terraform Cloud by introducing a custom, one-time migration step directly within a Bitbucket CI/CD pipeline.

This approach is particularly suited for teams that rely entirely on CI/CD pipelines to run Terraform. By performing the migration through the pipeline, the process remains consistent with existing workflows. 

In addition, in CI/CD environments, the machine executing Terraform runs is typically ephemeral and may not retain a .terraform directory between executions. As a result, Terraform may have no record of the previously used backend. In this scenario, when switching to Terraform Cloud, Terraform does not automatically attempt to migrate the existing state and instead initializes a new, empty state in the Terraform Cloud workspace.

Procedure

Modify your Terraform configuration to include a template file that can be used to overwrite the backend.tf file with the remote backend configuration. Then, create a custom CI/CD migration step that performs the following actions:

• Execute terraform init using the original backend configuration.

• Overwrite the Terraform backend configuration to use the remote (Terraform Cloud) backend.

• Execute terraform init -migrate-state -force-copy to migrate the existing state into Terraform Cloud.

Example: Migrating from Google Cloud Storage using BitBucket's CI/CD

1. Keep your original backend block inside a file called backend.tf.

terraform {
 backend "gcs" {
  bucket  = "..."
  prefix =  "..."
 }
}

2. Create the backend.remote.tf.tpl file with the remote backend block inside in the same folder.

terraform {
  backend "remote" {
    organization = "..."
    workspaces { name = "..." }
 }
}

3. Add a custom migrate_state step to your CI/CD pipeline and execute it once to perform the state migration, as shown in the example below.

custom:
    migrate_state:
      - step:
          name: One-time state migration (GCS → Terraform Cloud)
          script:
            - export GOOGLE_APPLICATION_CREDENTIALS={YOUR_GOOGLE_CREDENTIALS}
            - export TF_TOKEN_app_terraform_io="{YOUR_TERRAFORM_CLOUD_TOKEN}"

            # 1) Init against GCS (original source)
            - terraform init -input=false
            
            # 2) Verify the current state in GCS
            - terraform state list

            # 3) Switch backend type
            - cp backend.remote.tf.tpl backend.tf
            - terraform init -input=false -migrate-state -force-copy

            # 4) Verify state now lives in Terraform Cloud
            - terraform state list

During the migrate_state step, the first terraform init command ensures that Terraform is successfully initialized with the existing GCS backend. Next, the backend.remote.tf.tpl template is used to overwrite the active backend configuration, switching Terraform from the GCS backend to the remote (Terraform Cloud) backend. 

Finally, the second terraform init -input=false -migrate-state -force-copy command performs the actual state migration to Terraform Cloud. The -force-copy option suppresses the prompts and answers "yes" to the migration questions.

4. After the migration completes, commit the updated backend configuration (remote or cloud) to the repository. This ensures that future CI/CD runs consistently initialize Terraform against Terraform Cloud, rather than reverting to the previous backend

Expected Output

Initializing the backend...
Terraform detected that the backend type changed from "gcs" to "remote".
Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes. 

Additional Information

• https://developer.hashicorp.com/terraform/cli/commands/init#backend-initialization

• https://developer.hashicorp.com/terraform/language/backend/gcs