Terraform Enterprise FDO fails to start with EKS version >=1.30

Introduction

Problem

Terraform Enterprise FDO fails to start on EKS version >=1.30. During the start of the container the following logs show the error the S3 cannot be reached

panic: error loading config: failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded

goroutine 1 [running]:
main.main()
/home/runner/work/terraform-enterprise/terraform-enterprise/tools/service-setup/main.go:37 +0x34e

Prerequisites

• Terraform Enterprise FDO
• EKS (Kubernetes on AWS)
• Storage authentication is done using the instance profile

TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true

Cause

The issue happens because with the introduction of EKS version 1.30 AWS altered the default images for the Kubernetes nodes to Amazon Linux 2023 (AL2023). Please see the details here 

This has the side effect that there is a change regarding the IMDS configuration which is more strict now.

# get the instance details 
aws ec2 get-launch-template-data --instance-id <your eks worker instance>

"MetadataOptions": {
  "HttpTokens": "required",
  "HttpPutResponseHopLimit": 1,
  "HttpEndpoint": "enabled",
  "HttpProtocolIpv6": "disabled",
  "InstanceMetadataTags": "disabled"
},

The issue is the HttpPutResponseHopLimit which is and 1 and should be 2 for getting the correct credentials.  For a complete detail about IMDS see the following page here from the AWS documentation. 

A related to KB article to IMDS and Terraform Enterprise can be found here

Solutions:

Alter the launch template for your EKS nodes to have a HttpPutResponseHopLimit of 2. With Terraform code you would have something like the following 

resource "aws_launch_template" "k8-nodes" {
  name = "k8-nodes"

  # Instance Metadata Settings
  metadata_options {
    http_tokens = "required" 
    http_put_response_hop_limit = 2 
    http_endpoint = "enabled"
    http_protocol_ipv6 = "disabled" 
    instance_metadata_tags = "disabled" 
  }
}

resource "aws_eks_node_group" "k8s" {
  cluster_name = aws_eks_cluster.k8s.name
...
...
  launch_template {
     id = aws_launch_template.k8-nodes.id
     version = "$Latest"
  }
...
...
}

Outcome

The instances should show a HttpPutResponseHopLimit of 2

# get the instance details 
aws ec2 get-launch-template-data --instance-id <your eks worker instance>

"MetadataOptions": {
  "HttpTokens": "required",
  "HttpPutResponseHopLimit": 2,
  "HttpEndpoint": "enabled",
  "HttpProtocolIpv6": "disabled",
  "InstanceMetadataTags": "disabled"
},

Additional Information

• AWS documentation about EKS change can be found here

• IMDS documentation from AWS can be found here
• Related KB article about IMDS and Terraform Enterprise can be found here