Problem
When enabling log forwarding in an environment that has a proxy configured, logs aren't received by the log destination.
Prerequisites
- Terraform Enterprise, v202203-1 and later
Cause
-
NO_PROXY
doesn't contain a matching hostname entry for the log destination -
NO_PROXY
contains a partial hostname match for the log destination
To determine if the log forwarder is connecting successfully to the configured destination, check the log forwarding logs by connecting to the TFE host and running: docker logs tfe-fluent-bit
.
Connections that aren't successfully bypassing the proxy may produce errors similar to this:
2022-05-26T11:11:01.641380000Z [2022/05/26 11:11:01] [error] flb_establish_proxy error: ... Proxy response payload, usually an HTML error page ... 2022-05-26T11:11:01.709437000Z [2022/05/26 11:11:01] [ warn] [engine] failed to flush chunk '1-1653563460.253488461.flb', retry in 10 seconds: task_id=0, input=systemd.0 > output=splunk.0 (out_id=0)
Solution
-
Add the fully-qualified domain name for the destination to
NO_PROXY
- Restart the application
Outcome
Logs are successfully forwarded to the configured destination and the tfe-fluent-bit
container no longer logs any errors.
Additional Information
-
https://www.terraform.io/enterprise/admin/infrastructure/logging
- https://www.terraform.io/enterprise/install/interactive/installer#proxy-usage
- https://www.terraform.io/enterprise/install/interactive/installer#reconfiguring-the-proxy
- NO_PROXY matches only exact domain names #6759 (upstream issue in Fluent Bit)