Introduction
NOTE: Testing revealed this issue on AWS EC2 instances. It was found during testing of a deployment of the Boundary Worker in Docker
Support for Amazon Web Service’s IMDSv2 was added. Enabling IMDSv2 does require setting some additional configuration for Boundary client to not be negatively affected when mounted in a Docker container.
When this additional configuration is not set, but IMDSv2 is enabled, Boundary can experience a wide variety of symptoms. This missing configuration can present as general network issues, workspaces failing to complete runs with errors such as error #0, #3001, or RPC errors. Typically, enabling the proper configuration and redeploying the Boundary clients will resolve these issues if IMDSv2 has recently been enabled.
Please see IMDS-considerations in the AWS documentation.
Use Case
When enabling IMDSv2 on a Boundary Worker EC2 host, it is required to set the http-put-response-hop-limit option key to a value of 2 or greater. This meta-data option key limits the number of hops that metadata requests can travel across a network, and will affect the Worker’s ability to interact with S3 buckets. This is required due to the Boundary Enterprise, or Boundary HCP application being a set of containers running on the EC2 instance, making the minimum hop limit 2. Please note the exact setting required is entirely dependent on the environment that Boundary Enterprise, or Boundary HCP is deployed in. The maximum value for the http-put-response-hop-limit key is 64, but testing may be required to determine an exact value to set the key to.
Procedure
Enabling IMDSv2 on an instance requires use of the AWS CLI version 2. The AWS CLI will also need to be configured to connect to the AWS account and region where the Boundary Enterprise, or Boundary HCP instance is deployed.
Once AWS CLI version 2 has been configured, the only other piece of required information would be the Boundary Instance Id. Getting the Instance ID is usually easiest from the EC2 Service in the AWS Console, but can also be done by reviewing the output from the aws ec2 describe-instances command.
$ aws ec2 describe-instances
With the Instance ID for the Boundary Enterprise, or Boundary HCP, the AWS CLI modify-instance-metadata-optionscommand can now be used to enable/require IMDSv2, and set the http-put-response-hop-limit option key to the desired value.
Note that $BOUNDARY_INSTANCE_ID will need to be replaced with the target Instance ID for the Boundary Enterprise, or Boundary HCP Instance, and the value for http-put-response-hop-limit may need to be changed based on the instance’s environment.
$ aws ec2 modify-instance-metadata-options \
--instance-id $BOUNDARY_INSTANCE_ID \
--http-tokens required \
--http-endpoint enabled \
--http-put-response-hop-limit 2
The output will be similar to the following:
{
"InstanceId": "i-06e95fec9e1cc4ff4",
"InstanceMetadataOptions": {
"State": "pending",
"HttpTokens": "required",
"HttpPutResponseHopLimit": 2,
"HttpEndpoint": "enabled"
}
}
These settings can be verified using the following command and the State should eventually transition to applied.
aws ec2 describe-instances --instance-id $BOUNDARY_INSTANCE_ID
The command will return output that can be scrolled through, but will contain the following section showing the MetadataOptions and their settings.
MetadataOptions": {
"State": "applied",
"HttpTokens": "optional",
"HttpPutResponseHopLimit": 2,
"HttpEndpoint": "enabled"
}
Additional Information
Please note that when using an Amazon Web Services Auto Scaling Group, these settings would need to be included in the launch template for the ASG. This would allow these changes to persist in the event the current ASG EC2 instance is terminated and redeployed.