Problem
An incorrectly configured Web Application Firewall (WAF) may prevent Terraform Enterprise from functioning correctly. This can manifest as issues reaching object storage, saving or retrieving data, or operating workspace variables.
These issues often appear in the tfe_archivist container logs (or ptfe_archivist for versions before v202205-1) as unexpected HTTP 403, 404, or 413 error codes without a clear reason.
Prerequisites
- A Terraform Enterprise installation in a cloud environment (such as AWS, Azure, or GCP) that is protected by a WAF (such as AWS WAF, Azure WAF, or Google Cloud Armor).
Cause
Terraform Enterprise performs various internal operations by making API calls to itself. A WAF that is not configured to allow this internal traffic may incorrectly identify these legitimate requests as malicious and block them.
This can occur during several common operations:
- Uploading or downloading plans
- Uploading state files
- Saving variable values from the UI
- Accessing the application dashboard
Example Error Messages
The error messages can vary depending on the specific operation being blocked and the WAF configuration.
Nginx Logs
The tfe_nginx (or ptfe_nginx) container logs may show 404 errors for internal API calls.
10.16.33.16 - - [21/Jan/2020:20:22:54 +0000] "POST /app/gettyplus/workspaces/workspace-tfe-dev-09/runs HTTP/1.1" 404 5809 "-" "VSServices/16.179.30910.4 (w3wp.exe)"
Terraform UI or CLI
When saving state files, you may see a 403 Forbidden error.
Error uploading state: 403 Forbidden
Terraform Worker or Agent Logs
When uploading a plan, you may see a 413 error, indicating the request body is too large.
failed to upload plan json: Bad status code: 413
Note: The container names for Terraform Enterprise changed in version
v202205-1. The prefixptfe_(e.g.,ptfe_archivist) was changed totfe_(e.g.,tfe_archivist).
Solutions
To resolve this issue, you must configure the WAF to allow traffic required for Terraform Enterprise to operate.
Solution 1: Create WAF Exclusions for Terraform Enterprise
Create exclusion rules in your WAF to allow all traffic where the source and destination are the IP addresses of your Terraform Enterprise instances. This ensures that internal API calls are not blocked.
Solution 2: Create WAF Exclusions for Object Storage
In addition to allowing internal traffic, you may need to create an exclusion rule in the WAF for the object storage endpoint used by Terraform Enterprise. This ensures that Terraform Enterprise can reliably communicate with its storage backend.
Solution 3: Disable WAF for Terraform Enterprise Traffic
As a broader solution, you can disable the WAF entirely for all traffic to and from the Terraform Enterprise instance. While this is the simplest approach, it may not be suitable for all security postures. Evaluate this option based on your organization's security requirements.
Additional Information
- For detailed networking requirements for Terraform Enterprise, please refer to the official documentation.