Problem
When using the ServiceNow Service Catalog integration with Terraform Enterprise (TFE) , variables may be intermittently missing from connected workspaces. Reviewing the TFE support bundle logs reveals HTTP 429 Too Many Requests error codes in response to API calls from ServiceNow.
Example log entry:
[... TFE access log ...] "GET /api/v2/organizations/example-org/workspaces/example-workspace-1 HTTP/1.1" 429 119 "-" "ServiceNow/1.0" [... TFE access log ...] "GET /api/v2/organizations/example-org/workspaces/example-workspace-2 HTTP/1.1" 429 119 "-" "ServiceNow/1.0" [... TFE access log ...] "GET /api/v2/organizations/example-org/workspaces/example-workspace-3 HTTP/1.1" 429 119 "-" "ServiceNow/1.0"
Prerequisites
- A Terraform Enterprise instance configured with the ServiceNow Service Catalog integration.
Cause
By default, Terraform Enterprise enforces an API rate limit of 30 requests per second for both authenticated and unauthenticated requests. The HTTP 429 status code indicates that the ServiceNow integration is making too many API requests in a short period, exceeding this limit. When TFE throttles the requests, some operations fail, which can result in behaviors like missing variables in workspaces.
Solutions
Solution 1: Increase the API Rate Limit
You can resolve this issue by increasing the API rate limit within the Terraform Enterprise admin settings to accommodate the request volume from ServiceNow.
- Navigate to the Admin Settings page in your TFE instance.
- Select the API Rate Limiting section.
- Increase the value for Authenticated requests per second to a number that suits your environment's needs.
You can access this page directly at https://<your-tfe-hostname>/app/admin/settings.
Outcome
After increasing the API rate limit, the workspaces connected to the ServiceNow integration should no longer experience missing variables due to request throttling.