Overview
This article addresses the issue of encountering an
Expired Token
error during long-running apply runs in Terraform Cloud, particularly when deploying large infrastructure components such as RDS instances.
Problem
When running long Terraform apply operations, you may encounter the following error after approximately 1 hour:
This error indicates that the AWS security token used during the Terraform apply process has expired.
Cause
The error occurs because AWS service roles have a maximum session limit of 1 hour. During long-running operations, the session token may expire, leading to this error.
Solution
To resolve this issue, follow these steps:
1. Adjust Timeouts for AWS Resources
Terraform sets default timeouts for the creation, update, and deletion of AWS resources. For the
aws_db_instance
resource, the default timeouts are:
- Create: Default 40 minutes
- Update: Default 80 minutes
- Delete: Default 60 minutes
If your deployment involves creating large RDS instances, you may need to extend these timeouts. Here’s how to adjust the timeouts in your Terraform configuration:
2. Extend Session Duration in Provider Configuration
Ensure that your Terraform configuration uses an assume role with an extended session duration. Update the
duration_seconds
parameter in your provider configuration to match the maximum allowed by AWS (up to 12 hours):
provider "aws" {
region = var.region
default_tags {
tags = merge(
data.terraform_remote_state.outputs.outputs.default_tags,
{
Source = "Your-Source-Tag"
}
)
}
assume_role {
role_arn = local.cross_account_role_arn
session_duration = 7200 # 2 hours
duration_seconds = 7200 # 2 hours
}
}
3. Use AWS STS to Refresh Tokens
Implement a mechanism to refresh AWS STS tokens during long-running operations. This can be done by creating a custom script or using a tool that automatically refreshes the token before it expires. Here’s an example script:
#!/bin/bash
# Function to refresh AWS STS token
refresh_token() {
aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/YourRole" --role-session-name "terraform" > /tmp/aws_credentials.json
export AWS_ACCESS_KEY_ID=$(jq -r '.Credentials.AccessKeyId' /tmp/aws_credentials.json)
export AWS_SECRET_ACCESS_KEY=$(jq -r '.Credentials.SecretAccessKey' /tmp/aws_credentials.json)
export AWS_SESSION_TOKEN=$(jq -r '.Credentials.SessionToken' /tmp/aws_credentials.json)
}
# Refresh token every 45 minutes
while true; do
refresh_token
sleep 2700 # 45 minutes
4. Verify Role Configuration
Ensure that the role being used for the Terraform apply process has the necessary permissions and is configured correctly. Check the role configuration in your AWS Management Console and verify that it has the appropriate policies attached.
5. Use Terraform Cloud Workspaces with Long-Running Operations
Consider using Terraform Cloud workspaces that are specifically configured for long-running operations. This can help manage the session tokens and ensure that they are refreshed as needed.