How to download and analyze errored state files in Terraform Enterprise

Introduction

During a run performed by Terraform Enterprise (TFE), it may sometimes fail to upload a state file. When an apply fails, Terraform generates an errored state file in the TFE instance.  This file captures the state at the time the apply failed.  Its purpose is to help ensure that you don’t lose track of any changes made before the failure.

Expected Outcome

Download the errored state file, and review it. Based on that, decide whether to use it as the new source of truth or revert to the known good remote state.

Prerequisites

• Terraform Cloud Agent: >= 1.12.0

Procedure

1. Navigate to the run details where the apply failed. Copy the run ID from the URL or from the page header. It should look like: run-dsvgYHCTyhQtSEhV

2. Then, using the TFE API, get the apply ID from the run ID:

   curl -s \
     --header "Authorization: Bearer $TOKEN" \
     https://<TFE Host Name>/api/v2/runs/run-dsvgYHCTyhQtSEhV \
   | jq '.data.relationships.apply.data.id'

   Example output:

   "apply-Z2QqHj72m9wNa5wE"

3. Use that apply ID to retrieve the errored state:

   curl -s \
     --header "Authorization: Bearer $TOKEN" \
     https://$HOSTNAME/api/v2/applies/apply-Z2QqHj72m9wNa5wE/errored-state

   This request will return a temporary URL to download the errored state from.

4. Now compare the errored state file with the current state from the workspace, specifically focusing on the serial and lineage to check for any differences.  

• Serial: The serial number goes up with each successful state update. A higher serial usually means it's the most recent state.
• Lineage: Lineage is a unique ID for a specific state file history. If the lineage values are different, it means they’re from completely separate state histories.

Additional Information

• Applies API reference
• Manual State Pull/Push