Terraform Cloud/Enterprise API Not Returning All Data – HashiCorp Help Center

Problem

When you call the HCP Terraform or Terraform Enterprise API, the response may not include all of the expected data.

Cause

By default, the API paginates results and returns a maximum of 20 items per page. If a request generates more than 20 items, you must handle the pagination to retrieve the complete dataset.

Solutions

Solution 1: Increase the Page Size

You can increase the number of items returned per page by specifying the page[size] query parameter. The maximum page size is 100.

$ curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  https://app.terraform.io/api/v2/organizations/my-organization/workspaces?page%5Bsize%5D=100

Solution 2: Paginate Through Results with a Script

For results with a large number of items, you can use a script to loop through all pages until no more pages are available.

This example script uses jq to parse the next link from the API response and continues until the link is null.

#!/bin/bash
PAGE_NUMBER=1
ENDPOINT="https://app.terraform.io/api/v2/organizations/my-organization/workspaces"

while true; do
  RESPONSE=$(curl --silent --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/vnd.api+json" "$ENDPOINT?page%5Bnumber%5D=$PAGE_NUMBER")
  
  ## Process the response for the current page
  echo "$RESPONSE"

  ## Check if there are more pages
  NEXT_LINK=$(echo "$RESPONSE" | jq -r '.links.next')
  if [ "$NEXT_LINK" == "null" ]; then
    break
  fi

  ## Increment the page number for the next iteration
  PAGE_NUMBER=$((PAGE_NUMBER + 1))
done

Outcome

After applying one of these solutions, your API calls will return all data across one or more pages.

Additional Information

For more details on handling pagination, refer to the API Pagination Documentation.