Introduction
The use of dynamic provider credentials with Terraform Enterprise as described here states that the Cloud provider requires access to the Metadata Endpoints of your Terraform Enterprise environment.
If these endpoints are not available the workload identity is not able to verify the request and the authentication will fail.
Google Cloud provides the ability to upload this file manually as described here. This solves the requirement for the external access to metadata endpoints on your Terraform Enterprise configuration.
Expected Outcome
Using dynamic provider credentials for Google Cloud without the need to provide Terraform Enterprise the external access to metadata endpoints
Prerequisites
- Configure the Dynamic Provider Credentials following our documentation here
or - Use a Dynamic Provider Credentials tutorial as described here
Use Case
You have setup Dynamic Provider Credentials using the documentation and tutorial. Because the metedata endpoints of your Terraform Enterprise environment are not accessible from Google Cloud perspective you get the following error during a run
Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=hc-aaaaaaaaaaaa":
oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/tfc-service-account4@hc-aaaaaaaaaaaa.iam.gserviceaccount.com:generateAccessToken":
oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Error connecting to the given credential's issuer."}[0m
- Get the JWK file from your Terraform Enterprise environment
curl https://<tfe_FQDN>/.well-known/jwks > tfe_jwk.json
- On your Workload Identity Federation select the workload identity matching Terraform Enterprise and click edit
- Upload a new JWK file and select the one created earlier
- Authentication during a run should be successful
Extra notes
The JWK key on Terraform Enterprise will rotate automatically every 90 days. You can force the new key rotation yourself on which it will reset the timer. At a minimum you will have to upload the new key every 90 days or automate the new key generation and upload to Google Cloud on your own schedule.
Please see the steps for rotating the key here and trimming the key here.
There was a bug in Terraform Enterprise when there were more then 10 keys generated that workload identity didn't work properly anymore. This have been solved in version 202407-1 as documented here. Please look at trimming the key and the KB article specified here