Required Additional Configuration When Using IMDSv2

Introduction

NOTE: Testing revealed that the large majority of Terraform Enterprise instances were unaffected by this change. It was found during testing that a reboot of the Terraform Enterprise instance was required for the application to function correctly, and few required complete recreation of the instance.

Therefore, at this time enabling IMSDv2 is only suggested for those deployed in the External Services or Active-Active deployment modes.

With the release of Terraform Enterprise version v202103-1, release sequence 520, support for Amazon Web Service’s IMDSv2 was added. Enabling IMDSv2 does require setting some additional configuration in order for Terraform Enterprise to not be negatively affected.

When this additional configuration is not set, but IMDSv2 is enabled Terraform Enterprise can experience a wide variety of symptoms. This missing configuration can present as general network issues, workspaces failing to complete runs with errors such as SIC-001, or modules failing to ingress or publish changes. Typically enabling the proper configuration and redploying the Terraform Enterprise instance will resolve these issues if IMDSv2 has recently been enabled.

Use Case

When enabling IMDSv2 on a Terraform Enterprise EC2 host, it is required to set the http-put-response-hop-limit option key to a value of 2 or greater. This meta-data option key
limits the number of hops that metadata requests can travel across a network, and will affect Terraform Enterprise’s ability to interact with S3 buckets. This is required due to the Terraform Enterprise application being a set of containers running on the EC2 instance, making the minimum hop limit 2. Please note the exact setting required is entirely dependent on the environment that Terraform Enterprise is deployed in. The maximum value for the http-put-response-hop-limit key is 64, but testing may be required to determine an exact value to set the key to.

Procedure

Enabling IMDSv2 on an instance requires use of the AWS CLI version 2. The AWS CLI will also need to be configured to connect to the AWS account and region where the Terraform Enterprise instance is deployed.

Once AWS CLI version 2 has been configured, the only other piece of required information would be the Terraform Entprise Instance Id. Getting the Instance ID is usually easiest from the EC2 Service in the AWS Console, but can also be done by reviewing the output from the aws ec2 describe-instances command.

$ aws ec2 describe-instances

With the Instance ID for the Terraform Enterprise, the AWS CLI modify-instance-metadata-optionscommand can now be used to enable/require IMDSv2, and set the http-put-response-hop-limit option key to the desired value. Note that $TFE_INSTANCE_ID will need to be replaced with the target Instance ID for the Terraform Enterprise Instance, and the value for http-put-response-hop-limit may need to be changed based on the instance’s environment.

$ aws ec2 modify-instance-metadata-options \
    --instance-id $TFE_INSTANCE_ID \
    --http-tokens required \
    --http-endpoint enabled \
    --http-put-response-hop-limit 2

The output will be similar to the following:

{
    "InstanceId": "i-06e95fec9e1cc4ff4",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "enabled"
    }
}

These settings can be verified using the following command and the State should eventually transition to applied.

aws ec2 describe-instances --instance-id $TFE_INSTANCE_ID

The command will return output that can be scrolled through, but will contain the following section showing the MetadataOptions and their settigns.

MetadataOptions": {
                        "State": "applied",
                        "HttpTokens": "optional",
                        "HttpPutResponseHopLimit": 2,
                        "HttpEndpoint": "enabled"
                    }

Additional Information

Please note that when using an Amazon Web Services Auto Scaling Group, these settings would need to be included in the launch template for the ASG. This would allow these changes to persist in the event the current ASG EC2 instance is terminated and redeployed.