Terraform Enterprise FDO docker - External log forwarding - Amazon CloudWatch – HashiCorp Help Center

Introduction

By default, Docker uses the journald logging driver, so unless you set up something custom, the Terraform Enterprise container logs get wiped out whenever the container stops, restarts, or crashes.

We strongly recommend using an external log forwarding solution that aligns with your existing observability solutions. Depending on the deployment platform, native or third-party solutions (e.g., host-level monitoring agents) may be an appropriate solution for log aggregation and forwarding. HashiCorp does not provide support for third-party log forwarding solutions. (link)

This article has an example of how to use external log forwarding on Terraform Enterprise FDO Docker and forward Terraform Enterprise logs to Amazon CloudWatch.

Prerequisites

Terraform Enterprise FDO Docker hosted on AWS Cloud
Access to AWS CloudWatch
AWS IAM Role for the host with permissions to write on AWS CloudWatch (link)
Terraform Enterprise is managed as a systemd service (link)

Use Case

A log forwarding solution provides persistency on Terraform Enterprise logs after restart. This helps the Terraform Enterprise Admins and Organizations to have a centralized place for logs, including TFE Audit logs. It also makes troubleshooting easier since the logs from before a crash or restart are saved and can be reviewed.

Considerations

Using AWS services like CloudWatch comes with cost. It is the Terraform Enterprise administrator responsibility to set retentions policies and decide the CloudWatch configuration overall.
This guide does not set a retention policy, the logs will never expire unless they are deleted manually.

Procedure

Edit Terraform Enterprise docker compose yaml to use the Amazon CloudWatch Logs logging driver.
This change affects only the Terraform Enterprise docker container, it does not change the docker logging driver globally.

Connect to the Terraform Enterprise host with SSH.
Switch to root user, run:
```
sudo su -
```
Edit the docker-compose.yml and add the logging driver configuration under image:

logging:
      driver: "awslogs"
      options:
         awslogs-region: "<YOUR-AWS-REGION>"
         awslogs-group: "<A-NAME-FOR-TFE-LOGS-GROUP>"  
         awslogs-create-group: "true"
         tag: '{{- index (split .ImageName ":") 1 -}}_{{.Name}}_{{.ID}}'

Assuming the desired AWS region is eu-west-1 and the desired logs group name is tfe-node-1,
the docker compose file should like like this:

---
name: "terraform-enterprise"
services:
  tfe:
    image: "images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202501-1"
    logging:
      driver: "awslogs"
      options:
         awslogs-region: "eu-west-1"
         awslogs-group: "tfe-node-1"  
         awslogs-create-group: "true"
         tag: '{{- index (split .ImageName ":") 1 -}}_{{.Name}}_{{.ID}}' 
    environment:
...
...

Save the file

Restart Terraform Enterprise

systemctl restart terraform-enterprise.service

Terraform Enterprise should start without errors.

Check the logs on AWS CloudWatch

Open AWS web Console
Navigate to CloudWatch service
Under Logs click on Log groups (1)
In Log groups there should be a log group with name tfe-node-1 (2) as defined in the yaml above.
Click on tfe-node-1 Log group name

There should be a log stream with a name format of:
TFE_Version (1)_TFE_docker_container_name(2)_TFE_docker_container_id(3):

This naming format derives from the tag value that was used in the yaml file above. You can create your own tags or not use them at all.

tag: '{{- index (split .ImageName ":") 1 -}}_{{.Name}}_{{.ID}}'

With this tag format, a new log stream gets created whenever the Terraform Enterprise container is restarted or upgraded.

References