Problem
An incorrectly configured web application firewall may prevent Terraform Enterprise from reaching object storage, saving/retrieving data from it, or operating workspace variables. Often can be observed in Archivist logs container ptfe_archivist or tfe_archivist (For Terraform Enterprise v202205-1 or later) as HTTP error codes 403/404/413 of unknown origin and reason.
Prerequisites
- TFE installation in Azure, Amazon, or Google Cloud platform with WAF enabled.
Cause
NOTE: For Terraform Enterprise v202205-1 or later the container names have changed as the "p" has been dropped, such as for example pre v202205-1 container names would be ptfe_archivist and v202205-1 would resemble tfe_archivist.
- Performing various operations TFE is going to address its internals using the API. Often WAF will prevent such communications with a very enigmatic message
This can happen during :
- Plan uploading/downloading
- State uploading
- Saving of variable values from UI
- Even accessing the dashboard, in one documented case.
Some error messages examples:
- in ptfe_nginx or tfe_nginx logs :
10.16.33.16 - - [21/Jan/2020:20:22:54 +0000] "POST /app/gettyplus/workspaces/workspace-tfe-dev-09/runs HTTP/1.1" 404 5809 "-" "VSServices/16.179.30910.4 (w3wp.exe)""
- In TFE UI or TFE CLI with remote backend, during state saving :
Error uploading state: 403 Forbidden
- In TFE worker or agent logs :
failed to upload plan json: Bad status code: 413
Once more - please note that messages can vary, and it is highly dependable on your data and web application firewall settings. It is often confusing as there is no apparent reason.
Overview of possible solutions
- Disable WAF for TFE altogether
- Make exclusion in WAF for TFE IPs as source and destination
- In addition - you may need to make an exclusion in WAF for the utilized object storage endpoint