Introduction
Since release v202109-1, Terraform Enterprise (TFE) has included a Log Forwarding feature that enables TFE to forward its logs to one or more external destinations. This feature uses Fluent Bit, an open-source log processor and forwarder. This guide covers common troubleshooting steps for this feature.
Problem
After configuring the Log Forwarding feature in Terraform Enterprise, logs are not being sent to the external destination. The Fluent Bit container log may show configuration errors, or there may be no errors while logs still fail to arrive.
Cause
This issue can stem from two primary causes:
- The Fluent Bit configuration contains syntax errors or is not valid according to the required schema.
- The TFE instance cannot establish a network connection to the specified log forwarding endpoint due to firewall rules, proxy settings, or other network issues.
Solutions
Here are two common solutions to troubleshoot TFE log forwarding issues.
Solution 1: Validate the Fluent Bit Configuration
Ensure the Fluent Bit configuration syntax is valid by referencing the official TFE logging documentation and the Fluent Bit configuration schema.
An incorrect syntax will produce errors in the Fluent Bit log similar to the following.
[ Error] File /fluent-bit/etc/fluent-bit.conf [ Error] Error in line 19: Invalid indentation level Error: Configuration file contains errors. Aborting
Solution 2: Verify Log Endpoint Reachability
Confirm that the TFE instance can reach the log server endpoint with the current configuration. The specific tests depend on the Fluent Bit Output plugin you are using.
You can perform the following verification steps from within the TFE instance:
- Attempt to send a successful
curlrequest to the log server's HTTP endpoint. - Temporarily disable TLS-related configuration parameters to isolate certificate issues.
- If you use
HTTPS_PROXY,HTTP_PROXY, orNO_PROXYenvironment variables, ensure the log endpoint is correctly configured in these variables before starting TFE.