Problem
Logs forwarding is configured and enabled, but the logs are not being forwarded to the target system as intended. Additionally, container logs for the log forwarding container (tfe-fluent-bit) reveal no errors.
Prerequisites (if applicable)
- Terraform Enterprise, configured with log forwarding enabled
- Syslog
Cause
The log_forwarding_config has an invalid configuration for the syslog_message_key.
name syslog
match *
host syslog.logserver.com
port 514
mode udp
syslog_format rfc5424
syslog_maxsize 2048
syslog_hostname_key hostname
syslog_appname_key appname
syslog_procid_key procid
syslog_msgid_key msgid
syslog_sd_key uls@0
syslog_message_key log
Overview of possible solutions (if applicable)
Solutions:
1. Review the current log_forwarding_configby running the following command:
replicatedctl app-config export --template '{{.log_forwarding_config.Value}}'
[OUTPUT]Name syslogMatch *host syslog.logserver.comport 514mode udpsyslog_message_key logsyslog_severity_key PRIORITYsyslog_hostname_key _HOSTNAMEsyslog_appname_key SYSLOG_IDENTIFIERsyslog_procid_key _PID
2. Update the fluent-bit.conf file ensuring that thesyslog_message_key value is set to message:
fluent-bit.conf
[OUTPUT]Name syslogMatch *host syslog.logserver.comport 514mode udpsyslog_message_key messagesyslog_severity_key PRIORITYsyslog_hostname_key _HOSTNAMEsyslog_appname_key SYSLOG_IDENTIFIERsyslog_procid_key _PID
3. Update the log_forwarding_config by running the following command on each node:
replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"
4. Restart the application for the changes to take effect.
replicatedctl app stop
replicatedctl app start
Outcome
Logs will be forwarded as intended to the syslog host. If issues persist, please contact HashiCorp Support for further assistance.
Additional Information
- https://docs.fluentbit.io/manual/pipeline/outputs/syslog