Terraform Enterprise Resource Recovery: Leveraging Taint, Untaint, and Replace

Introduction

When managing infrastructure with Terraform, cloud resources can sometimes drift from their intended state due to manual changes, corruption, or other unexpected behavior. In these situations, you may need to recreate a specific resource without destroying and reapplying the entire configuration. This article explains how to use Terraform's lifecycle management capabilities to selectively recreate unhealthy or misconfigured resources.

Problem

A cloud resource managed by Terraform has become unhealthy, degraded, or misconfigured. For example, a virtual machine may exhibit poor performance or security vulnerabilities not reflected in the configuration.

Running terraform apply does not resolve the issue because Terraform does not detect any changes in the configuration files and assumes the existing resource is still valid. This can leave you with a problematic resource in your environment.

Prerequisites

This guide uses a sample configuration with two AWS EC2 instances to demonstrate the procedures. You can adapt the resource addresses to match your own configuration.

Example configuration:

resource "aws_instance" "my_vm_1" {
  ami           = var.ami ## e.g. Ubuntu AMI
  instance_type = var.instance_type

  tags = {
    Name = "VM 1"
  }
}

resource "aws_instance" "my_vm_2" {
  ami           = var.ami
  instance_type = var.instance_type

  tags = {
    Name = "VM 2"
  }
}

Solutions

Terraform provides two primary methods for forcing the replacement of a resource. The recommended approach is to use the -replace flag with the apply command.

Solution 1: Use the apply -replace Flag (Recommended)

The terraform apply -replace command allows you to force the recreation of a specific resource in a single, atomic operation. This is the modern and safest method, as it avoids modifying the state file prior to the apply run.

To replace aws_instance.my_vm_2, run the following command. Terraform will plan and apply the replacement in one step.

$ terraform apply -replace="aws_instance.my_vm_2"

In Terraform Enterprise or HCP Terraform

In Terraform Enterprise and HCP Terraform, runs execute remotely, so you cannot pass command-line flags directly. Instead, you can use a workspace environment variable to pass the -replace option to the remote run.

1. Navigate to your workspace and select the Variables page.
2. Under Environment Variables, select Add variable.
3. Set the Key to TF_CLI_ARGS_plan.
4. Set the Value to -replace=aws_instance.my_vm_1 (or the address of the resource you want to replace).
5. Ensure the Environment Variable option is selected and save the variable.
6. Queue a new plan. The plan output will show that the specified resource is scheduled for destruction and recreation.
7. Approve and apply the run to complete the replacement.

This process achieves the same outcome as running the command locally but operates entirely within the remote execution environment.

Solution 2: Use the terraform taint Command (Deprecated)

Note: The terraform taint command is deprecated as of Terraform v0.15.2. You should use the apply -replace flag instead.

The taint command marks a resource in the Terraform state file as "tainted," flagging it for destruction and recreation on the next apply.

1. Mark the resource as tainted. For example, to taint aws_instance.my_vm_1, run the following command.

   $ terraform taint aws_instance.my_vm_1

   Terraform confirms the action.

   Resource instance aws_instance.my_vm_1 has been marked as tainted.

2. Run terraform plan to review the changes. The plan will show that the tainted resource will be destroyed and recreated.
3. Run terraform apply to execute the plan.

If you taint a resource by mistake, you can reverse the action using the terraform untaint command before running apply.

Caution: The taint command modifies the state file immediately. If another team member initiates a run after you have tainted a resource but before you have applied the change, the resource may be replaced unintentionally.

Outcome

By using the -replace flag or the deprecated taint command, you can exercise fine-grained control over your infrastructure's lifecycle. This allows you to resolve issues with specific resources, such as corruption or misconfiguration, without affecting the rest of your infrastructure, thereby minimizing downtime and maintaining stability.

Additional Information

• For more details on the deprecated taint command, refer to the Terraform CLI taint command documentation.