How to download and analyze errored state files in Terraform Enterprise

Introduction

When an apply run fails in Terraform Enterprise (TFE), Terraform generates an errored state file. This file captures the state at the moment the apply failed. This file helps ensure that you do not lose track of any resource changes made before the failure.

This guide explains how to download and analyze the errored state file to determine the next steps for your workspace.

Expected Outcome

You will download the errored state file from a failed TFE run and review its contents. After your review, you can decide whether to use it as the new source of truth or revert to the last known good remote state.

Prerequisites

• Terraform Enterprise instance
• API token with permissions to read runs for the workspace
• Terraform Cloud Agent version 1.12.0 or newer

Procedure

Follow these steps to download and inspect the errored state file.

1. Locate the Run ID

   Navigate to the run details page for the failed apply. Copy the run ID from the URL or the page header. It will have a prefix of run-.

   Example run ID: run-dsvgYHCTyhQtSEhV

2. Retrieve the Apply ID

   Use the TFE API and the run ID to retrieve the corresponding apply ID. Replace <TFE_HOSTNAME> with your TFE instance's hostname and provide your API token.

   $ curl -s \
     --header "Authorization: Bearer $TOKEN" \
     https://<TFE_HOSTNAME>/api/v2/runs/run-dsvgYHCTyhQtSEhV | \
     jq '.data.relationships.apply.data.id'

   The command returns the apply ID.

   "apply-Z2QqHj72m9wNa5wE"

3. Retrieve the Errored State File URL

   Use the apply ID from the previous step to request the errored state file. This request returns a temporary, pre-signed URL that you can use to download the file.

   $ curl -s \
     --header "Authorization: Bearer $TOKEN" \
     https://<TFE_HOSTNAME>/api/v2/applies/apply-Z2QqHj72m9wNa5wE/errored-state

4. Download and Analyze the State File

   Use the URL from the previous step's output to download the errored state file. Compare this file with the current state from the workspace. You can download the current state from the workspace's States tab in the TFE UI.

   When comparing the files, focus on the serial and lineage values to check for differences.

   • serial: The serial number increments with each successful state update. A higher serial number usually indicates a more recent state.
   • lineage: This value is a unique ID for a specific state file's history. If the lineage values are different, the state files are from completely separate histories.

Additional Information

• For more details on the API endpoint, refer to the Applies API reference.
• To learn more about manually managing state, see the documentation on Manual State Pull/Push.