Fluentd Agent fails to start after the upgrade to fluent-package v5 – HashiCorp Help Center

Introduction

Problem

Fluentd Agent fails to start after the upgrade from td-agent v4 to fluent-package v5

Please note that Fluentd is an open source data collector for unified logging layer which is not supported by HashiCorp Technical Support. The content of this article is informational only and has been obtained from the respective Fluentd Github repositories.

Prerequisites (if applicable)

Cause

The /var/log/fluent/fluentd.log contains:

2025-01-27 14:10:19 +0000 [warn]: [output_td] Use different plugin for secondary. Check the plugin works with primary like secondary_file primary="Fluent::Plugin::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2025-01-27 14:10:19 +0000 [error]: config error file="/etc/fluent/fluentd.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'splunk_hec'. Run 'gem search -rd fluent-plugin' to find plugins"

This error message can be caused by one of the following reasons:

The fluent-plugin-splunk-enterprise Plugin isn't installed
The fluent-plugin-splunk-hec Plugin has been uninstalled automatically during the upgrade of the upgrade from td-agent v4 to fluent-package v5
Both the fluent-plugin-splunk-enterprise Plugin and the fluent-plugin-splunk-hec Plugin are installed.
The fluentd configuration file has to be modified after the upgrade from td-agent v4 to fluent-package v5

Overview of possible solutions (if applicable)

Solutions:

Solution 1
Install the fluent-plugin-splunk-enterprise Plugin using:
"fluent-gem install fluent-plugin-splunk-enterprise" and make sure to update the fluentd configuration file as the agent fails to start with the migrated configuration, please find the example configuration below:

<match vault_audit.**>
@type splunk_hec
host splunkserver.domain.com
port 8088
token *********-****-****-****-***********
</match>

Solution 2
Reinstall the fluent-plugin-splunk-hec Plugin using:
"fluent-gem uninstall fluent-plugin-splunk-hec" and make sure the fluentd configuration file looks as follows:

<match vault_audit.**>
@type splunk_hec
hec_host splunkserver.domain.com
hec_port 8088
hec_token *********-****-****-****-***********
</match>

The fluent-plugin-splunk-hec github repository contains the following disclaimer:

The fluent-plugin-splunk-hec will reach End of Support on January 1, 2024. After that date, this repository will no longer receive updates from Splunk and will no longer be supported by Splunk. Until then, only critical security fixes and bug fixes will be provided.

Solution 3
Make sure that either one of the plugins is installed, the following command can be used to list the installed plugins:

fluent-gem list |grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-elasticsearch (5.4.3)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.3)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.4)
fluent-plugin-prometheus (2.1.0)
fluent-plugin-prometheus_pushgateway (0.1.1)
fluent-plugin-record-modifier (2.2.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.8.1)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-splunk-enterprise (0.10.2)
fluent-plugin-systemd (1.1.0)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.6.0)

One of the plugins can be uninstalled as follows:

sudo fluent-gem uninstall fluent-plugin-splunk-hec

sudo fluent-gem uninstall fluent-plugin-splunk-enterprise

Outcome

Both the fluent-plugin-splunk-enterprise Plugin and the fluent-plugin-splunk-hec Plugin are HTTP Event Collector plugins, which are used to send events to Splunk using the HEC format.
The expected outcome is that the fluent-package v5 is able to start and eventually parse files generated by the Vault Audit Device.

Additional Information