ExpiredToken Errors During Long-Running Apply Runs in Terraform Cloud

Problem

During a long-running terraform apply in HCP Terraform, you may encounter an ExpiredToken error after approximately one hour, especially when provisioning large infrastructure components like Amazon RDS database instances. The error message may resemble the following.

Error: Error creating DB Instance: ExpiredToken: The security token included in the request is expired

Cause

This error occurs because the temporary AWS security token used by Terraform through an assumed IAM role has a default session duration of one hour. If the Terraform operation takes longer than this, the token expires, causing AWS API calls to fail.

Solutions

To resolve this issue, you can apply one or more of the following solutions within your Terraform configuration.

Solution 1: Adjust AWS Resource Timeouts

For resources that take a long time to provision, you can extend the default timeouts within the resource block. For example, the aws_db_instance resource has default timeouts that may be insufficient for large instances.

You can override these timeouts in your configuration. This example extends the creation timeout to two hours.

resource "aws_db_instance" "default" {
  # ... other configuration ...

  timeouts {
    create = "120m"
  }
}

Solution 2: Extend the AWS Assumed Role Session Duration

You can increase the session duration for the assumed IAM role in your AWS provider configuration. The maximum duration is 12 hours (43200 seconds).

Update the duration argument in the assume_role block. Note that the duration_seconds argument is deprecated.

provider "aws" {
  region = var.region

  assume_role {
    role_arn = "arn:aws:iam::123456789012:role/YourRole"
    duration = "2h" ## Set duration up to 12 hours (e.g., "12h")
  }
}

Solution 3: Verify IAM Role Permissions

Ensure the IAM role that HCP Terraform assumes has the necessary permissions to perform all required actions. Missing permissions can sometimes cause delays or failures that extend the run time. Review the IAM role's trust policy and attached permission policies in the AWS Management Console to confirm they are configured correctly.

Outcome

After applying these configuration changes, HCP Terraform runs should be able to complete long-running operations without encountering ExpiredToken errors, as the session token will remain valid for the extended duration.