Introduction
This article provides troubleshooting steps for an issue where log forwarding in Terraform Enterprise fails when a proxy is configured.
Problem
When you enable log forwarding in a Terraform Enterprise environment that has a proxy configured, logs are not received by the log destination.
Prerequisites
- Terraform Enterprise v202203-1 and later
Cause
The issue may be caused by one of the following configuration problems in the NO_PROXY environment variable:
- The
NO_PROXYvariable does not contain a matching hostname entry for the log destination. - The
NO_PROXYvariable contains only a partial hostname match for the log destination, which is insufficient.
To determine if the log forwarder is connecting successfully to the configured destination, you can check the log forwarding logs. Connect to the Terraform Enterprise host and run the following command.
$ docker logs tfe-fluent-bit
Connections that do not successfully bypass the proxy may produce errors similar to the following output, which often includes an HTML error page from the proxy.
[error] flb_establish_proxy error: ... Proxy response payload, usually an HTML error page ... [warn] [engine] failed to flush chunk '1-1653563460.253488461.flb', retry in 10 seconds: task_id=0, input=systemd.0 > output=splunk.0 (out_id=0)
Solution
To resolve this issue, you must ensure the log destination's domain is explicitly excluded from the proxy.
- Add the fully-qualified domain name (FQDN) for the log destination to the
NO_PROXYenvironment variable. - Restart the Terraform Enterprise application for the changes to take effect.
Outcome
After applying the solution, logs should be successfully forwarded to the configured destination, and the tfe-fluent-bit container should no longer produce connection errors.