Log forwarding using Fluent-bit in Terraform Enterprise in Google Cloud – HashiCorp Help Center

Introduction

Terraform Enterprise supports forwarding its logs to one or more external destinations, a process called log forwarding. Log forwarding provides increased observability, assistance complying with log retention requirements, and information during troubleshooting.

In this article, we are majorly focusing on forwarding the logs to Google Cloud Logging in Terraform Enterprise deployed in replicated mode.

Expected Outcome

The logs will be forwarded to external system which again can be enriched as per the requirement by other monitoring tools.

Prerequisites

Terraform Enterprise host admin access.
Adequate GCP credentials roles having access to Cloud logging.
Terraform Enterprise running on an instance using systemd-journald. Execute
```
systemctl status systemd-journald
```
to check if the systemd-journald service is started and enabled.
A version of Docker that supports the journald logging driver. Execute
```
docker info --format '{{.Plugins.Log}}'
```
to check if the journald plugin is listed.
Network connectivity between Terraform Enterprise and the external destination(s) where logs should be forwarded.

Use Case

The article illustrates the procedure for configuring log forwarding in replicated mode of deployment.

Procedure

Run the below command to enable log forwarding in TFE.

$ tfe-admin app-config -k log_forwarding_enabled -v 1

Validate the replicate settings for log forwarding:

$ replicatedctl app-config export --hidden | grep -A2 log_forwarding_enabled 

Output:
--------
"log_forwarding_enabled": { 
"value": "1" 
},
--------

Create the external destinations to forward the logs. For this create fluent-bit.conf file as per your requirements. Sample configurations are available here:

Content for GCP Cloud Forwarding:

[OUTPUT]
    Name       stackdriver                                      ### Plug-in Name (For GCP Cloud logger, stackdriver####
    Match      *
    location   us-west1                                         ### Region where TFE is hosted
    namespace  terraform_enterprise                             #### Any name #####
    node_id    <FQDN>                                           ### Hostname/IP for the TFE instance ####
    resource   generic_node

Note: In Terraform Enterprise installations using GCP external services, Fluent Bit will have access to the GOOGLE_SERVICE_CREDENTIALS environment variables that points to a attribute gcs_credentials of replicated app settings.

When configured,gcs_credentials is also used for object storage access by Terraform Enterprise. So make sure that the gcs_credentials must have the required access to the object storage also.

For this, add the gcs_credentials into replicated settings using below command:

$ replicatedctl app-config set gcs_credentials --value “$(cat connector.key.json)”

Verify the gcs_credentials are available in replicated app settings using the below command:

$ replicatedctl app-config export --hidden | grep -A2 gcs

References:

https://developer.hashicorp.com/terraform/enterprise/replicated/monitoring/logging