Unable to update workspace allow list for an agent pool

Problem

When attempting to update the list of allowed workspaces for an agent pool in Terraform Enterprise, the operation fails with the following error, even when the UI shows no workspaces are connected.

Error saving agent pool
Agent pool is still being used by workspaces in your organization

Attempts to update the agent pool using the API or the TFE provider also fail with a 422 Unprocessable Entity error.

Cause

This issue occurs when workspaces that were previously connected to the agent pool were deleted on Terraform Enterprise version v202208-1 or older. In these versions, a background job that should remove the deleted workspace after 30 minutes fails to complete, leaving behind remnant data that still associates the deleted workspace with the agent pool.

While Terraform Enterprise versions after v202208-1 delete workspaces correctly, versions prior to v202308-1 cannot update an agent pool if these remnants from older deleted workspaces are present.

An API call to inspect the agent pool will not show these remnant workspaces, making them difficult to identify.

{
  "data": {
    "id": "apool-pSJHZQ3TDq6qiC14",
    "type": "agent-pools",
    "attributes": {
      "name": "agent_pool",
      ## ...
    },
    "workspaces": {
      "data": []
    }
  }
  ## ...
}

Solutions

There are three approaches to resolve this issue. The recommended solution is to upgrade Terraform Enterprise. If an upgrade is not immediately possible, you may use one of the provided workarounds.

Solution 1: Upgrade Terraform Enterprise

Upgrade your Terraform Enterprise instance to version v202309-1 (build 733) or higher. In these versions, the agent pool update process correctly ignores remnant workspace data, allowing you to save changes successfully.

Solution 2: Create a New Agent Pool

A temporary workaround is to create a new agent pool and associate the required workspaces with it. This allows you to continue operations until you can perform the recommended upgrade.

Solution 3: Manually Disconnect Workspaces via Rails Console

This workaround involves accessing the Terraform Enterprise host to manually disassociate the remnant workspaces from the agent pool. This procedure should only be performed if you are unable to upgrade.

1. Establish an SSH session into your Terraform Enterprise host.

2. Connect to the Rails console.

   $ sudo docker exec -it tfe-atlas /usr/bin/init.sh /app/scripts/wait-for-token -- bash -i -c 'cd /app && ./bin/rails c'

3. Find your agent pool using its external ID (e.g., apool-pSJHZQ3TDq6qiC14) and note its internal id.

   AgentPool.find_by_external_id("apool-pSJHZQ3TDq6qiC14")

   The output will show the internal id.

   => #<AgentPool id: 5, external_id: "apool-pSJHZQ3TDq6qiC14", ...organization_scoped: false>

4. Using the internal id from the previous step (e.g., 5), retrieve all workspaces associated with the agent pool.

   Workspace.where('agent_pool_id = 5')

   The output will list all associated workspaces, including the remnants.

   [#<Workspace:0x00007f625dd7e5d0
   id: 602,
   name: "test-ws-xtp9x",
   external_id: "ws-hbbTzPogiyopdwGR",
   organization_id: 2,
   #...
   agent_pool_id: 5,
   execution_mode: "agent",
   #...
   discarded_at: Mon, 31 Jul 2023 16:07:38.883200000 UTC +00:00,
   #...

5. Identify the remnant workspaces by comparing the list from the Rails console with the workspaces visible in the Terraform Enterprise UI. The remnants will appear in Rails but not in the UI. Confirm that these remnants have a discarded_at timestamp older than 30 minutes.

6. For each remnant workspace, use its internal id (e.g., 602) to disconnect it from the agent pool and reset its execution mode.

   Workspace.update(602, "agent_pool_id" => nil, "execution_mode" => "remote")

   Repeat this step for all identified remnant workspaces.

Outcome

After applying one of the solutions, you will be able to successfully update the allowed workspaces list for the agent pool.