Problem
All workspaces using Dynamic Provider Credentials to authenticate providers begin failing with token signature verification errors. Sometimes user encounter issue in their environment due to a issue in the code related to the automatic rotation of signing keys. This issue causes the key rotation mechanism to incorrectly identify the newest signing key. Resulting in error messages like InvalidIdentityToken: Token signature invalid or failed to verify
id token signature.
The following are examples of errors which may manifest, depending on the cloud provider in use.
AWS
Error: failed to refresh cached credentials, failed to retrieve credentials,exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: abcde123-a234-567d-abcde9874621, InvalidIdentityToken: Token signature invalid
Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found. Please see https://registry.terraform.io/providers/hashicorp/aws for more information about providing credentials. Error: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Token signature invalid status code: 400, request id: bdc94a4b-d21e-4be0-9973-d6d3f03fc1b8
Vault
Operation failed: failed authenticating to Vault: Error making API request. URL: PUT https://your_vault_address.example.com:8200/v1/auth/jwt/tf/login Code: 400. Errors: * error validating token: error verifying token signature: failed to verify id token signature
Azure
Error: building account: could not acquire access token to parse claims: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS7000274: Key 43ef235f1a1674828b08c529de1f1f1ffb1f4ff73f5f123456ba565dfc8509c4 was found, but use of the key to verify the signature failed. Trace ID: 12345678-a4a3-5552-001f-03132bd43c00 Correlation ID: 87123456-a0a1-23aa-92fd-bc24b9718c41 Timestamp: 2024-07-08 01:10:01Z","error_codes":[7000274],"timestamp":"2024-07-08 01:10:01Z","trace_id":"a6f8c29c-a60a-4446-991e-08132bd33a21","correlation_id":"cee699aa-0c6a-34cb-42ea-bc42d6268048","error_uri":"https://login.microsoftonline.com/error?code=7000274"}
GCP
oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{ACCOUNT_EMAIL_OR_UNIQUEID}:generateAccessToken": oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."}
Kubernetes
Error: Unauthorized
Prerequisites
- Terraform Enterprise v202207-1 through v202406-1
- Terraform Enterprise >= v202207-1 has been installed for ~810 days with the default 90 day OIDC signing key rotation period
Cause
OIDC signing keys are rotated periodically and utilize a monotonically increasing integer to denote key versions.
These versions were sorted lexicographically in one code path, but numerically in another. This caused an issue when multiple key versions exist that have a different number of digits and a previous key version is selected for use.
Solution
HCP Terraform
The issue has been fixed and is not expected to cause further impact.
Terraform Enterprise
The issue will manifest in all versions of TFE prior to v202407-1 that support Dynamic Credentials, after the OIDC signing key is rotated 9 times. The default key rotation period for OIDC signing keys is 90 days, meaning all workspaces utilizing Dynamic Credentials will start failing approximately 810 days after the first installation of a version of TFE that supports Dynamic Credentials. The first version of TFE that supported Dynamic Credentials is v202207-1 and this means that instances upgraded to version 202207-1 close to its release can start to see failures during October of 2024.
Upgrade TFE to Latest Version
This issue was fixed in v202407-1 of TFE. Upgrading is the recommended solution to the issue.
Forcefully Rotate and Trim Signing Keys
Using the Rails console, the issue can be temporarily resolved by rotating the OIDC signing key and trimming historic versions:
# rotate key 9 times
9.times { OIDC::KeyManager.instance.rotate_key }
# perform trim operation to trim all keys except the latest
OIDC::KeyManager.instance.trim_key
This is not a permanent solution, but depending on the configured key rotation period (default 90 days), it will allow for safe rotations of the key until the 100th rotation, at which point the issue will recur.
Extend Key Rotation Period
If the issue has yet to be observed, upgrading is not an option, and manually triggering rotation is not an option, the key rotation period can be extended to delay occurrence of the issue.
This key rotation can be configured via the WORKLOAD_IDENTITY_AUTO_ROTATE_PERIOD environment variable. The value is the number of days to wait between automatic rotations of the OIDC signing key, extending the time it will take to rotate to the 10th key version. This period should remain as short as possible, but a key rotation period of a year or more should be perfectly safe.