Introduction
Forwarding logs from Terraform Enterprise (TFE) to Splunk enables robust monitoring and analysis of your TFE environment. By integrating Splunk with TFE, you can gain deep insights into your Terraform runs, track system performance, and proactively identify issues. This guide will walk you through the steps to configure TFE to send its logs to Splunk, ensuring that you can leverage Splunk's powerful search, analysis, and visualization capabilities to manage your Terraform infrastructure effectively.
Expected Outcome
- Terraform Enterprise (TFE) logs will be successfully forwarded to Splunk.
- You will have the capability to monitor and analyze TFE logs in real-time using Splunk's powerful search and analysis tools.
Prerequisites (if applicable)
- TFE setup should be FDO-Docker
- Should have Splunk account with index created to collect the logs
Use Case
The goal is to send Terraform Enterprise (TFE) logs to Splunk for enhanced monitoring and analysis.
Procedure
Step 1-: Login to the TFE host machine and create the Splunk configuration file
Note: Adding the [FILTER] section and "event_host" to the splunk.conf file allows you to tag the messages coming from TFE with a hostname for more detailed filtering inside of Splunk.
#vi splunk.conf
#Example syntax
root@ip-172-31-37-12:~/fdo-test# cat splunk.conf
[FILTER]
Name modify
Match *
Add hostname <name of host>[OUTPUT]
Name splunk
Match *
Host *******.splunkcloud.com
Port 8088
Splunk_Token *********
tls On
tls.verify Off
event_host hostname
Step 2-: Amend the existing compose.yaml file by incorporating the provided content, or alternatively, generate a new compose.yaml file and include the following information.
Note - Take the reference of below configuration and modify as per current environment
#Observability settings. See the configuration reference for more settings.
TFE_LOG_FORWARDING_CONFIG_PATH: /var/tmp/splunk_config/splunk.conf
TFE_LOG_FORWARDING_ENABLED: "true"
TFE_METRICS_ENABLE: "true"
TFE_METRICS_HTTP_PORT: "9090"
TFE_METRICS_HTTPS_PORT: "9091"
and
type: bind
source: /root/fdo-test
target: /var/tmp/splunk_config
TFE_LOG_FORWARDING_CONFIG_PATH : refers to the path inside the container from which TFE will retrieve the Splunk configuration.
Source: This indicates the location on your TFE host where you've stored the Splunk configuration file.
Target: It should match the location specified by TFE_LOG_FORWARDING_CONFIG_PATH.
Step 3-: Initiate the TFE (Terraform Enterprise) instance and oversee its error-free initialisation. Establish connection with the TFE container. Afterward, confirm the output of the following command:
tfectl app config | grep "log_forwarding" -A4
"log_forwarding": {
"ConfigData": "[OUTPUT]\n Name splunk\n Match *\n Host example-splunk-hec-endpoint\n Port 8088\n Splunk_Token example-splunk-token\n\n",
"config_path": "/var/tmp/splunk_config/splunk.conf",
"enabled": true
},
Step 4-: Now verify the logs in the splunk in created index