Introduction
Problem
Terraform Enterprise is configured to have a ServiceNow Integration for automated provisioning of the resources.The workspace connected to ServiceNow Integration is missing some variables and this behaviour is intermittent.
Checking the Support Bundle's logs, the 429 response error code is raised:
2023-12-04T22:24:11.792282842Z 172.24.60.149 - - [04/Dec/2023:22:24:11 +0000] "GET /api/v2/organizations/daniela-org/workspaces/test-workspace1 HTTP/1.1" 429 119 "-" "ServiceNow/1.0"
2023-12-04T22:24:11.797893710Z 172.24.60.149 - - [04/Dec/2023:22:24:11 +0000] "GET /api/v2/organizations/daniela-org/workspaces/test-workspace2 HTTP/1.1" 429 119 "-" "ServiceNow/1.0"
2023-12-04T22:24:11.807593529Z 172.24.60.148 - - [04/Dec/2023:22:24:11 +0000] "GET /api/v2/organizations/daniela-org/workspaces/test-workspace3 HTTP/1.1" 429 119 "-" "ServiceNow/1.0"
Prerequisites
- TFE is using ServiceNow Service Catalog Integration
Cause
- Terraform Enterprise has by default a rate limit of 30 requests per second to the API as an authenticated or unauthenticated request.
- The rate limit has been reached as per the 429 status code from the logs and there are too many requests done to TFE and the access is throttled, hence intermittent behaviour of missing variables can be observed
Overview of possible solutions
Solutions:
-
Increase the API Rate Limit from TFE's Admin Settings (User > Admin > Settings > API Rate Limiting): https://<your-tfe-hostname>/app/admin/settings
Outcome
The workspaces with ServiceNow Integration no longer have any missing variables.