No logs received from TFE log forwarding

Problem

Log forwarding is configured and enabled in Terraform Enterprise, but logs are not being forwarded to the target system. Additionally, logs for the tfe-fluent-bit container do not show any errors.

Prerequisites

• Terraform Enterprise with log forwarding enabled.
• A configured Syslog target.

Cause

The log_forwarding_config contains an invalid value for the syslog_message_key parameter. The correct key is message, but it has been set to log.

Example of an incorrect configuration:

name syslog
match *
host syslog.logserver.com
port 514
mode udp
syslog_format rfc5424
syslog_maxsize 2048
syslog_hostname_key hostname
syslog_appname_key appname
syslog_procid_key procid
syslog_msgid_key msgid 
syslog_sd_key uls@0
syslog_message_key log

Solution

Follow these steps to correct the log_forwarding_config value on each Terraform Enterprise node.

1. Review the current log_forwarding_config by running the following command.

   $ replicatedctl app-config export --template '{{.log_forwarding_config.Value}}'
   
   [OUTPUT] Name syslog Match * host syslog.logserver.com port 514 mode udp syslog_message_key log syslog_severity_key PRIORITY syslog_hostname_key _HOSTNAME syslog_appname_key SYSLOG_IDENTIFIER syslog_procid_key _PID

2. Create a file named fluent-bit.conf with the corrected configuration, ensuring the syslog_message_key is set to message.

   [OUTPUT]
   Name syslog
   Match *
   host syslog.logserver.com
   port 514
   mode udp
   syslog_message_key message
   syslog_severity_key PRIORITY
   syslog_hostname_key _HOSTNAME
   syslog_appname_key SYSLOG_IDENTIFIER
   syslog_procid_key _PID

3. Update the log_forwarding_config using the file you just created.

   $ replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"

4. Restart the application for the changes to take effect.

   $ replicatedctl app stop

   $ replicatedctl app start

Outcome

After the application restarts, logs will be forwarded as intended to the syslog host.

Additional Information

• Terraform Enterprise Syslog Configuration
• Fluent Bit Syslog Output Documentation