Introduction
Expected Outcome
Enable Log Forwarding and confirm it is successful in Terraform Enterprise.
Prerequisites
- Terraform Enterprise v202109-1 or later
Use Case
- Customer is configuring Log Forwarding on Terraform Enterprise but needs further assistance after following the official documentation.
Procedure
-
Before proceeding determine the type of installation, active/active or standalone(external-services-online or mounted-disk).
- Create a
fluent-bit.conffile with the[OUTPUT]section as outlined in the documentation for supported external destinations.
- For active/active installs there are two approaches.
- Option 1 is to drain the secondary node and once this process has completed scale in your ASG from 2 to 1 so that you can perform all the changes in one node and then scale out.
- Option 2 is to make the changes on each node at a time after draining it and then restarting the TFE app. All of these changes are performed in the CLI.
- For standalone installs changes can be applied via the replicated console or cli. For Terraform Enterprise flexible deployment options installations, the container or pod needs to be re-deployed after the YAML file has been updated.
- After the changes, to verify that fluent-bit is successfully processing the records, make a call to the fluent-bit API to check for metrics
- List of commands.
# Standalone Installs
replicatedctl app-config set log_forwarding_enabled --value 1
replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"
replicatedctl app apply-config # Restarts TFE to apply new config if changes have been made
# Active/Active Installs
tfe-admin node-drain # Node stops receiving new connections and completes existing tasks
tfe-admin app-config -k log_forwarding_enabled -v 1
tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"
replicatedctl app stop
watch replicatedctl app status # Confirm the APP has successfully stopped
replicatedctl app start
# For Terraform Enterprise flexible deployment options refer to https://developer.hashicorp.com/terraform/enterprise/flexible-deployments/install/configuration#observability-settings
# docker-compose.yaml settings
# Log Forwarding
TFE_LOG_FORWARDING_CONFIG_PATH: "/container/path/to/fluent-bit.conf"
TFE_LOG_FORWARDING_ENABLED: true
# Use the fluent-bit API to verify the records are being processed
curl -s http://$(docker inspect tfe-fluent-bit|jq -r '.[].NetworkSettings.Networks[].IPAddress'):2020/api/v1/metrics|jq
# If jq is not available
curl -s http://$(docker inspect tfe-fluent-bit -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'):2020/api/v1/metrics
# For Terraform Enterprise flexible deployment options. NOTE: Container name might differ
docker exec -it terraform-enterprise-tfe-1 curl -s http://localhost:2020/api/v1/metrics|jq
- API call output examples. Failure vs Success.