Problem
Terraform Enterprise Flexible Deployment Options (FDO) fails to start on Amazon Elastic Kubernetes Service (EKS) version 1.30 or newer. During container startup, the logs show an error indicating that the S3 bucket cannot be reached.
panic: error loading config: failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded goroutine 1 [running]: main.main()/home/runner/work/terraform-enterprise/terraform-enterprise/tools/service-setup/main.go:37 +0x34e
Prerequisites
- Terraform Enterprise Flexible Deployment Options (FDO).
- An Amazon EKS cluster version 1.30 or newer.
- Terraform Enterprise is configured to use an instance profile for S3 object storage authentication.
TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true
Cause
With the release of EKS version 1.30, AWS updated the default Amazon Machine Images (AMIs) for Kubernetes nodes to Amazon Linux 2023 (AL2023). This update introduced a stricter default configuration for the Instance Metadata Service (IMDS), specifically setting the HttpPutResponseHopLimit to 1.
You can verify this setting by inspecting the launch template data for a worker node.
$ aws ec2 get-launch-template-data --instance-id <your-eks-worker-instance>
Example output:
{
"LaunchTemplateData": {
"MetadataOptions": {
"HttpTokens": "required",
"HttpPutResponseHopLimit": 1,
"HttpEndpoint": "enabled",
"HttpProtocolIpv6": "disabled",
"InstanceMetadataTags": "disabled"
}
}
}Terraform Enterprise requires a hop limit of 2 to retrieve credentials from the instance metadata endpoint. The default value of 1 prevents the container from accessing the credentials, causing the S3 connection to fail.
Solution
To resolve this issue, you must modify the launch template for your EKS worker nodes to set the HttpPutResponseHopLimit to 2.
If you manage your EKS infrastructure with Terraform, update your aws_launch_template resource with the correct metadata_options.
resource "aws_launch_template" "k8-nodes" {
name = "k8-nodes"
# Instance Metadata Settings
metadata_options {
http_tokens = "required"
http_put_response_hop_limit = 2
http_endpoint = "enabled"
http_protocol_ipv6 = "disabled"
instance_metadata_tags = "disabled"
}
}
resource "aws_eks_node_group" "k8s" {
cluster_name = aws_eks_cluster.k8s.name
# ...
launch_template {
id = aws_launch_template.k8-nodes.id
version = "$Latest"
}
# ...
}After applying this change, your EKS nodes will be recreated with the updated IMDS configuration, allowing Terraform Enterprise to start successfully.
Outcome
After updating the launch template, verify that new instances have a HttpPutResponseHopLimit of 2.
$ aws ec2 get-launch-template-data --instance-id <your-new-eks-worker-instance>
Example output:
{
"LaunchTemplateData": {
"MetadataOptions": {
"HttpTokens": "required",
"HttpPutResponseHopLimit": 2,
"HttpEndpoint": "enabled",
"HttpProtocolIpv6": "disabled",
"InstanceMetadataTags": "disabled"
}
}
}