Log forwarding using Fluent-bit in Terraform Enterprise in Google Cloud (Replicated)

Introduction

Terraform Enterprise supports forwarding its logs to one or more external destinations. This process, known as log forwarding, provides increased observability, helps comply with log retention requirements, and provides critical information during troubleshooting.

This article provides the procedure for configuring log forwarding to Google Cloud Logging for a Terraform Enterprise instance deployed in Replicated mode.

Expected Outcome

After completing this procedure, Terraform Enterprise application logs will be forwarded to your configured Google Cloud Logging destination.

Prerequisites

Before you begin, you must have the following:

• Administrative access to the Terraform Enterprise host.
• Google Cloud Platform (GCP) credentials with roles and permissions sufficient to write to Cloud Logging.

• Terraform Enterprise running on an instance that uses systemd-journald. To verify, execute the following command.

  $ systemctl status systemd-journald

• A version of Docker that supports the journald logging driver. To verify, execute the following command.

  $ docker info --format '{{.Plugins.Log}}'

  The output should include journald.

• Network connectivity between the Terraform Enterprise instance and the Google Cloud Logging service endpoint.

Procedure

Follow these steps to configure log forwarding to Google Cloud Logging.

1. Enable the log forwarding feature in Terraform Enterprise.

   $ tfe-admin app-config -k log_forwarding_enabled -v 1

2. Validate that the setting was applied correctly.

   $ replicatedctl app-config export --hidden | grep -A2 log_forwarding_enabled

   The output should show a value of 1.

   ##... 
   "log_forwarding_enabled": {
     "value": "1"
   },
   ##...

3. Create a fluent-bit.conf file to define the external log destination. Sample configurations are available in the official documentation for Supported External Destinations.

   For this use case, create the file with the following content.

   [OUTPUT]
       Name        stackdriver # Plug-in Name (For GCP Cloud logger, use stackdriver)
       Match       *
       location    us-west1    # Region where TFE is hosted
       namespace   terraform_enterprise # Any descriptive name
       node_id     <FQDN>      # Hostname/IP for the TFE instance
       resource    generic_node

4. Configure Terraform Enterprise with the necessary GCP credentials. In Terraform Enterprise installations that use GCP external services, Fluent Bit can access the GOOGLE_SERVICE_CREDENTIALS environment variable. This variable points to the gcs_credentials attribute in the Replicated application settings.

   Note: The gcs_credentials attribute is also used for object storage access by Terraform Enterprise. Ensure the service account key you provide has the required permissions for both Google Cloud Logging and your GCS object storage bucket.

   Set the gcs_credentials value using your GCP service account key file (e.g., connector.key.json).

   $ replicatedctl app-config set gcs_credentials --value "$(cat connector.key.json)"

5. Verify that the gcs_credentials are present in the Replicated application settings.

   $ replicatedctl app-config export --hidden | grep -A2 gcs

Additional Information

• For more details on logging configurations and supported destinations, refer to the Terraform Enterprise Monitoring and Logging documentation.