Problem
Incorrectly configured web application firewall may prevent Terraform Enterprise from reaching object storage, saving/retrieving data from it, or operate workspace variables. Often can be observed in Archivist logs (container `ptfe_archivist*`) as HTTP error codes 403/404/413 of unknown origin and reason.
Prerequisites
- TFE installation in Azure, Amazon, or Google Cloud platform with WAF enabled.
Cause
- Performing various operations TFE is going to address its internals using the API. Often WAF will prevent such communications with a very enigmatic message
This can happen during :
- Plan uploading/downloading
- State uploading
- Saving of variable values from UI
- Even accessing the dashboard, in one documented case.
Some error messages examples:
- in `ptfe_nginx` logs :
10.16.33.16 - - [21/Jan/2020:20:22:54 +0000] "POST /app/gettyplus/workspaces/workspace-tfe-dev-09/runs HTTP/1.1" 404 5809 "-" "VSServices/16.179.30910.4 (w3wp.exe)""
- In TFE UI or TFE CLI with remote backend, during state saving :
Error uploading state: 403 Forbidden
- In TFE `worker` or agent logs :
failed to upload plan json: Bad status code: 413
Once more - please note that messages can vary, and it is highly dependable on your data and web application firewall settings. It is often confusing as there is no apparent reason.
Overview of possible solutions
- Disable WAF for TFE altogether
- Make exclusion in WAF for TFE IPs as source and destination
- In addition - you may need to make exclusion in WAF for utilized object storage endpoint