How to use Dynamic Provider Credentials on GCP without External Access to Metadata Endpoints – HashiCorp Help Center

Introduction

The use of dynamic provider credentials with Terraform Enterprise for Google Cloud Platform (GCP) requires that the cloud provider can access the metadata endpoints of your Terraform Enterprise instance. If these endpoints are not externally accessible, workload identity cannot verify the request, and authentication will fail.

This guide provides a procedure to configure dynamic provider credentials without exposing your Terraform Enterprise metadata endpoints. It uses a GCP feature that allows you to manually upload the OpenID Connect (OIDC) JSON Web Key (JWK) configuration file, as described in the GCP documentation.

Expected Outcome

You will be able to use dynamic provider credentials for GCP with Terraform Enterprise without providing external access to the instance's metadata endpoints.

Prerequisites

You have configured Dynamic Provider Credentials for GCP by following the official GCP configuration documentation or the Dynamic Credentials tutorial.

Use Case

When Terraform Enterprise metadata endpoints are not accessible from Google Cloud, runs may fail with the following error:

Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=hc-aaaaaaaaaaaa": oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/tfc-service-account4@hc-aaaaaaaaaaaa.iam.gserviceaccount.com:generateAccessToken": oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Error connecting to the given credential's issuer."}

Procedure

Follow these steps to manually upload the JWK file to your GCP Workload Identity Federation configuration.

Download the JWK file from your Terraform Enterprise instance. Replace <tfe_FQDN> with your instance's fully qualified domain name.
```
$ curl https://<tfe_FQDN>/.well-known/jwks > tfe_jwk.json
```
In the Google Cloud Console, navigate to Workload Identity Federation and select the workload identity pool that corresponds to your Terraform Enterprise instance.
Click EDIT to modify the provider configuration.
In the provider configuration, select UPLOAD JWKS and choose the tfe_jwk.json file you downloaded in the first step.

After uploading the file, authentication during a Terraform run should succeed.

Important Considerations

Key Rotation

The JWK signing key in Terraform Enterprise rotates automatically every 90 days. You must re-upload the new key to GCP at least every 90 days to maintain authentication. You can also automate this process or force a key rotation on your own schedule using the Terraform Enterprise API.

Product Bug Fix

A bug existed in older versions of Terraform Enterprise where workload identity would fail if more than 10 keys were present. This issue was resolved in version v202407-1, as noted in the release notes. For more details, refer to the KB article on the OIDC Vault key rotation issue.

Additional Information