AWS provider 4.0 "no valid credential sources for Terraform AWS Provider found" problem in AWS Cloud – HashiCorp Help Center

Problem

After an intentional or accidental upgrade of AWS Provider to version 4.0 you encounter the following error in the run.

no valid credential sources for Terraform AWS Provider found

While valid credentials were provided.

Prerequisites

You are using the AWS Provider 4.0
Within runs initiated from an EC2 instance - in Terraform Enterprise, Terraform Cloud, or Terraform Cloud agents that are executing on EC2 instances.

Cause

AWS provider could not connect to EC2 metadata server for authentication due to missing hop http-put-response-hop-limit (required by IMDSv2) - e.g. it can be applied only to some instances.

Overview of possible solutions

Downgrade to AWS provider release 3.74 or any previously used and working release.
Modify affected EC2 instance metadata and increase http-put-response-hop-limit See instruction below.

How to increase HttpPutResponseHopLimit

Enabling IMDSv2 on an instance requires the use of the AWS CLI version 2. The AWS CLI will also need to be configured to connect to the AWS account and region where the Terraform Enterprise instance/TF Cloud Agent is deployed.

Once AWS CLI version 2 has been configured, the only other piece of required information would be the Terraform Enterprise INSTANCE_ID. Getting the INSTANCE_ID is usually easiest from the EC2 Service in the AWS Console, but can also be done by reviewing the output from the aws ec2 describe-instances command.

$ aws ec2 describe-instances

With the INSTANCE_ID for the Terraform Enterprise, the AWS CLI modify-instance-metadata-options command can now be used to enable/require IMDSv2, and set the http-put-response-hop-limit option key to the desired value. Note that $TFE_INSTANCE_ID will need to be replaced with the target INSTANCE_ID for the Terraform Enterprise Instance, and the value for http-put-response-hop-limit may need to be changed based on the instance’s environment.

$ aws ec2 modify-instance-metadata-options \
    --instance-id $TFE_INSTANCE_ID \
    --http-tokens required \
    --http-endpoint enabled \
    --http-put-response-hop-limit 2

The output will be similar to the following:

{
    "InstanceId": "i-06e95fec9e1cc4ff4",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "enabled"
    }
}

These settings can be verified using the following command and the State should eventually transition to applied.

aws ec2 describe-instances --instance-id $TFE_INSTANCE_ID

The command will return output that can be scrolled through but will contain the following section showing the MetadataOptions and their settings.

MetadataOptions": {
                        "State": "applied",
                        "HttpTokens": "optional",
                        "HttpPutResponseHopLimit": 2,
                        "HttpEndpoint": "enabled"
                    }

After instance metadata HttpPutResponseHopLimit was increased runs should continue without errors.

Additional Information

Original article about IMDSv2 problem with HTTP hops applicable for TFE can be found here: https://support.hashicorp.com/hc/en-us/articles/1500005343862-Required-Additional-Configuration-When-Using-IMDSv2