Gitlab : The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.

Problem

The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.

Server responded with code 401, message: Token was revoked. You have to re-authorize from the user.

Cause

There is a race condition where if Gitlab receives a lot of webhooks at once then it will refresh the token on each webhook which then revokes the other token(s) and also invalidates the VCS connection.

This can be verified by reviewing the Gitlab production logs and searching for /oauth/token posts. In the example below, you can see that a refresh token request was posted and succeeded with a 200 but this process took Gitlab 60 seconds and within that time another token refresh request was created and failed since the primary request was still in progress and the previous token was invalidated.

Example:

gitlab-logs-2020-07-02/production_json_all.log:{"method":"POST","path":"/oauth/token","format":"*/*","controller":"Doorkeeper::TokensController","action":"create","status":200,"duration":60.17,"view":0.0,"time":"2020-07-02T12:59:09.108Z","params":[{"key":"client_id","value":""},{"key":"client_secret","value":"[FILTERED]"},{"key":"grant_type","value":"refresh_token"},{"key":"refresh_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"qyYcaGBHM94"}

gitlab-logs-2020-07-02/production_json_all.log:{"method":"POST","path":"/oauth/token","format":"*/*","controller":"Doorkeeper::TokensController","action":"create","status":401,"duration":3.43,"view":0.0,"time":"2020-07-02T12:59:58.626Z","params":[{"key":"client_id","value":""},{"key":"client_secret","value":"[FILTERED]"},{"key":"grant_type","value":"refresh_token"},{"key":"refresh_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"dKsHi6Ej3t2"}

Solution

To restore VCS access you’ll need to revoke and re-establish the VCS connection but this will require you to reconnect the workspaces.

Additional Information

To avoid this race condition in the future it is advisable to recreate the VCS connection through the TFE provider or using the API. When creating a VCS connection through the API or the provider, it generates a personal access token and Terraform Enterprise can use this token in place of the Oauth token. The personal access token should not need to be refreshed since this token is generated by the user. It should remain valid until you invalidate it.

API: https://www.terraform.io/docs/cloud/api/oauth-clients.html#create-an-oauth-client
TFE Provider: https://registry.terraform.io/providers/hashicorp/tfe/latest/docs/resources/oauth_client

Please see https://gitlab.com/gitlab-org/gitlab/-/issues/243844 for more details.