Additional AWS EC2 Configuration Required When Using IMDSv2 on Terraform Enterprise Servers

Overview

Terraform Enterprise version v202103-1 (release sequence 520) and later support Amazon Web Service’s IMDSv2. To ensure Terraform Enterprise functions correctly, you must apply additional configuration when you enable IMDSv2.

Important: At this time, HashiCorp suggests enabling IMDSv2 only for Terraform Enterprise instances deployed in External Services or Active-Active modes. While most instances are unaffected, some may require a reboot or recreation to function correctly after this change.

Problem

Enabling IMDSv2 without the required configuration can cause a variety of symptoms, including general network issues, failed workspace runs with errors like SIC-001, or modules failing to ingress or publish changes. This occurs because the Terraform Enterprise application runs in containers on the EC2 instance, which affects how metadata requests travel across the network to services like S3.

Setting the http-put-response-hop-limit to 2 or greater resolves this by accommodating the extra network hop from the container to the instance metadata service.

Prerequisites

Before you begin, you need the following:

• AWS CLI version 2 installed.
• The AWS CLI configured for the AWS account and region where the Terraform Enterprise instance is deployed.
• The Instance ID of your Terraform Enterprise EC2 instance. You can find this in the EC2 Service in the AWS Console.

Procedure

Follow these steps to modify the instance metadata options for your Terraform Enterprise instance.

1. Set an environment variable for your Terraform Enterprise instance ID to simplify the commands. Replace i-0123456789abcdef0 with your actual instance ID.

   $ export TFE_INSTANCE_ID="i-0123456789abcdef0"

2. Run the modify-instance-metadata-options command to require IMDSv2 and set the hop limit. The value for http-put-response-hop-limit may need to be higher than 2 depending on your network environment, but it cannot exceed 64.

   $ aws ec2 modify-instance-metadata-options \
     --instance-id $TFE_INSTANCE_ID \
     --http-tokens required \
     --http-endpoint enabled \
     --http-put-response-hop-limit 2

3. The command returns output indicating the change is pending.

   {
     "InstanceId": "i-06e95fec9e1cc4ff4",
     "InstanceMetadataOptions": {
       "State": "pending",
       "HttpTokens": "required",
       "HttpPutResponseHopLimit": 2,
       "HttpEndpoint": "enabled"
     }
   }

4. Verify that the settings have been applied. The State should transition from pending to applied.

   $ aws ec2 describe-instances --instance-id $TFE_INSTANCE_ID

5. Review the command output and locate the MetadataOptions section to confirm the state is applied.

   ## ...
   "MetadataOptions": {
     "State": "applied",
     "HttpTokens": "required",
     "HttpPutResponseHopLimit": 2,
     "HttpEndpoint": "enabled"
   }
   ## ...

Applying the proper configuration and redeploying the Terraform Enterprise instance typically resolves any issues that occurred after enabling IMDSv2.

Important Considerations

If you use an Amazon Web Services Auto Scaling Group (ASG) for your Terraform Enterprise deployment, you must include these metadata settings in the ASG's launch template. This ensures the configuration persists if the EC2 instance is terminated and redeployed by the ASG.