Problem
When using Gitlab as a VCS provider, users may encounter an error when attempting to ingress from VCS within a workspace. The following error is also displayed in the ptfe_atlas & ptfe_sidekiq containers (in Terraform Enterprise 202205-1 these containers are called tfe-atlas and tfe-sidekiq). VCS access on the workspace is lost.
The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.
Server responded with code 401, message: Token was revoked. You have to re-authorize from the user.
Cause
There is a known race condition where if Gitlab receives a lot of webhooks at once then it will refresh the token on each webhook which then revokes the other token(s) and also invalidates the VCS connection.
This can be verified by reviewing the Gitlab production logs and searching for /oauth/token posts. In the example below, you can see that a refresh token request is posted and succeeded with a 200, but this process took Gitlab 60 seconds. Within that time another token refresh request is created, and failed since the primary request is still in progres. The previous token is then invalidated.
Example:
gitlab-logs-2020-07-02/production_json_all.log:{"method":"POST","path":"/oauth/token","format":"*/*","controller":"Doorkeeper::TokensController","action":"create","status":200,"duration":60.17,"view":0.0,"time":"2020-07-02T12:59:09.108Z","params":[{"key":"client_id","value":""},{"key":"client_secret","value":"[FILTERED]"},{"key":"grant_type","value":"refresh_token"},{"key":"refresh_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"qyYcaGBHM94"}
gitlab-logs-2020-07-02/production_json_all.log:{"method":"POST","path":"/oauth/token","format":"*/*","controller":"Doorkeeper::TokensController","action":"create","status":401,"duration":3.43,"view":0.0,"time":"2020-07-02T12:59:58.626Z","params":[{"key":"client_id","value":""},{"key":"client_secret","value":"[FILTERED]"},{"key":"grant_type","value":"refresh_token"},{"key":"refresh_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"dKsHi6Ej3t2"}Solution
To restore VCS access the token needs to be revoked and re-established in the VCS connection. This will also require the workspaces to be connected to the VCS connection.
Additional Information
To avoid this race condition in the future, it is advisable to recreate the VCS connection through the TFE provider or using the API. When creating a VCS connection through the API or the provider, a personal access token is generated. Terraform Enterprise can then use this personal access token in place of the Oauth token. The personal access token would not need to be refreshed since this token is generated by the user, and will remain valid until manually invalidated.
API: https://www.terraform.io/docs/cloud/api/oauth-clients.html#create-an-oauth-client
TFE Provider: https://registry.terraform.io/providers/hashicorp/tfe/latest/docs/resources/oauth_client