Google API Error 429 Quota exceeded for 'Queries' and limit reached for 'Queries per minute per user'

Problem

When using the Google Cloud Platform (GCP) provider to provision a large set of infrastructure, you may encounter a 429 Too Many Requests error. This can occur when frequent Terraform runs, such as automatic speculative plans in HCP Terraform, exceed your Google API quota for Queries per minute per user.

This example refers to the google_dns_managed_zone data source, but this error may occur with other GCP resources.

A terraform plan may produce the following error output.

on .terraform/modules/example/data.tf line 6, in data "google_dns_managed_zone" "example_zone": 6: data "google_dns_managed_zone" "example_zone" { Error: googleapi: Error 429: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'dns.googleapis.com' for consumer 'project_number:111123456789'., rateLimitExceeded

Enabling trace logging with TF_LOG=TRACE may show the following API response.

{
  "error": {
    "code": 429,
    "message": "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'dns.googleapis.com' for consumer 'project_number:111123456789'.",
    "errors": [
      {
        "message": "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'dns.googleapis.com' for consumer 'project_number:111123456789'.",
        "domain": "global",
        "reason": "rateLimitExceeded"
      }
    ],
    "status": "RESOURCE_EXHAUSTED"
  }
}

Cause

This error is caused by a Google Cloud Platform API limitation. The default quota for requests per minute per user is often insufficient for configurations with a large number of resources that are refreshed during every Terraform run.

Solutions

HashiCorp recommends reporting feedback to Google about the requests per minute per user limit if it impacts your workflows.

Solution 1: Separate Long-Lived Resources into Workspaces

To reduce the number of API calls during each run, separate long-lived or seldom-updated resources into their own configurations and workspaces. For example, a database, once provisioned, is rarely modified. Moving its configuration to a dedicated workspace prevents Terraform from refreshing its state during runs that manage other, more dynamic infrastructure. This frees up available API requests for the resources that change frequently.

Solution 2: Disable Automatic Speculative Plans

If you are using HCP Terraform with a version control system (VCS) connection, you can disable automatic speculative plans. This setting prevents HCP Terraform from automatically running a terraform plan on every commit to a pull request, which can consume a large number of API calls.

1. Navigate to your workspace.
2. Go to Settings > Version Control.
3. Disable Automatic speculative plans.

Note that disabling this feature means you must manually queue plans to review infrastructure changes before applying them.

Solution 3: Increase Time Between Runs

A straightforward workaround is to wait for a longer period between executing Terraform runs. This allows your per-minute API request quota to reset, providing more capacity for the next run.

Solution 4: Use Multiple GCP Service Accounts

Create multiple GCP provider instances in your Terraform configuration, each using different service account credentials. By distributing API requests across multiple accounts, you can effectively multiply the total number of available requests per minute.

Additional Information

• For more details on Google Cloud Platform API quotas, refer to the official GCP: Usage Limits documentation.
• For guidance on organizing your infrastructure, review the documentation on HCP Terraform Workspaces.