GitLab Token Unable To Refresh Due To Race Condition

Problem

When using GitLab as a VCS provider with Terraform Enterprise, you may encounter an error when attempting to ingress from VCS within a workspace, causing the workspace to lose its VCS connection.

The following error appears in the ptfe_atlas and ptfe_sidekiq containers (named tfe-atlas and tfe-sidekiq in Terraform Enterprise v202205-1 and later).

The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.
Server responded with code 401, message: Token was revoked. You have to re-authorize from the user.

Cause

A known race condition can occur if your GitLab instance receives a high volume of webhooks simultaneously. GitLab attempts to refresh the OAuth token for each webhook, which revokes the previously issued token. This rapid succession of token refreshes and revocations invalidates the VCS connection in Terraform Enterprise.

You can verify this by reviewing the GitLab production logs for /oauth/token POST requests. In the following example, the first token refresh request succeeds after 60 seconds. However, a second refresh request during that time fails with a 401 error because the original token was invalidated by the first request before it completed.

{
  "method": "POST",
  "path": "/oauth/token",
  "controller": "Doorkeeper::TokensController",
  "action": "create",
  "status": 200,
  "duration": 60.17,
  "time": "...",
  "params": [
    {
      "key": "client_id",
      "value": ""
    },
    {
      "key": "client_secret",
      "value": "[FILTERED]"
    },
    {
      "key": "grant_type",
      "value": "refresh_token"
    },
    {
      "key": "refresh_token",
      "value": "[FILTERED]"
    }
  ],
  "correlation_id": "qyYcaGBHM94"
}

{
  "method": "POST",
  "path": "/oauth/token",
  "controller": "Doorkeeper::TokensController",
  "action": "create",
  "status": 401,
  "duration": 3.43,
  "time": "...",
  "params": [
    {
      "key": "client_id",
      "value": ""
    },
    {
      "key": "client_secret",
      "value": "[FILTERED]"
    },
    {
      "key": "grant_type",
      "value": "refresh_token"
    },
    {
      "key": "refresh_token",
      "value": "[FILTERED]"
    }
  ],
  "correlation_id": "dKsHi6Ej3t2"
}

Solutions

There are two solutions to this issue. The first solution provides an immediate fix to restore access, while the second solution is the recommended long-term approach to prevent the race condition from recurring.

Solution 1: Restore VCS Access Manually

To restore VCS access immediately, you must revoke the existing token and re-establish the VCS connection in your Terraform Enterprise organization settings. After re-establishing the connection, you must reconnect the affected workspaces to the VCS provider.

Solution 2: Prevent the Race Condition with a Personal Access Token

To avoid this race condition in the future, recreate the VCS connection using a GitLab Personal Access Token (PAT) instead of an OAuth token. A PAT is generated by a user and remains valid until manually revoked, so it is not subject to the refresh race condition.

You can create the VCS connection programmatically using its PAT with either the HCP Terraform API or the tfe provider.

• HCP Terraform API: Refer to the Create an OAuth Client API documentation for instructions on creating a VCS connection via the API.
• Terraform tfe Provider: Use the tfe_oauth_client resource to manage the VCS connection as a Terraform resource.