Introduction
This document will explain when a Google API Error 429 may occur and how to bypass it. In this example, we will refer to the Data Source "google_dns_managed_zone", but this error may occur for other configurations.
Problem
While using the Google Cloud Platform (GCP) provider with Service Account credentials to provision a large set of infrastructure using Terraform, automatic triggering of Speculative Plans are exceeding your Google API quota for Queries per minute per user. This quota can be found under the GCP “Quotas” section of your Google dashboard and you may encounter the error even if the maximum limit of 1200 is set. See GCP: Usage Limits for more details.
Terraform Plan output:
on .terraform/modules/example/data.tf line 6, in data "google_dns_managed_zone" "example_zone": 6: data "google_dns_managed_zone" "example_zone" {
Error: googleapi: Error 429: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'dns.googleapis.com' for consumer 'project_number:111123456789'., rateLimitExceeded
TF_LOG=TRACE output:
[DEBUG] Google API Response Details: ---[ RESPONSE ]--------------------------------------
HTTP/2.0 429
Too Many Requests
...
{ "error": { "code": 429, "message": "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'dns.googleapis.com' for consumer 'project_number:111123456789'.",
"errors": [ { "message": "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'dns.googleapis.com' for consumer 'project_number:111123456789'.",
"domain": "global", "reason": "rateLimitExceeded" } ], "status": "RESOURCE_EXHAUSTED" } }
-----------------------------------------------------:
Cause
This is a limitation of Google’s 1200 maximum requests per minute per user.
Solutions
It’s recommended to report to Google any feedback you have about the maximum
requests per minute per user.
Best Option
Determine if the number of resources within the same configuration can be separated in a logical way. For example, once a Database is provisioned it is seldom updated. This makes Database configuration a good candidate for its own Workspace. Separating this will avoid unnecessary API calls and refreshes freeing up more requests per minute per user. Overall if there are objects you do not change often, we recommend giving them their own Workspaces.
Temporary Option
If you are using TFC or TFE and have a VCS connection, you can go to
Your Workspace > Settings > Version Control > and disable
Automatic speculative plans.
This option is not always ideal since Speculative Plans help to mitigate risk by allowing users to review configuration before applying.
Performance Impact Option
Another workaround is to wait a longer period of time between executing runs. This allows more time for requests per minute per user to free up.
Additional Setup Option
Create more GCP provider instances with different credentials and accounts.