Audit logs are missing from Terraform Enterprise – HashiCorp Help Center

Problem

Audit log entries intermittently disappear from Terraform Enterprise logs. This can affect any component, but this article uses a Sentinel policy check as an example.

For a successful Sentinel run, you should see three distinct audit log entries: created, queued, and passed.

[INFO] [Audit Log] {"resource":"policy_check","action":"created"...}
[INFO] [Audit Log] {"resource":"policy_check","action":"queued"...}
[INFO] [Audit Log] {"resource":"policy_check","action":"passed"...}

However, in affected environments, only a subset of these logs may appear, such as the queued entry.

[INFO] [Audit Log] {"resource":"policy_check","action":"queued","resource_id":"polchk-3ucnF1gM4if7nPfK"...}

Prerequisites

Terraform Enterprise version older than v202402-1.

Cause

The Terraform Enterprise application logs are processed by fluent-bit. If a log entry exceeds the default buffer size of 32KB, fluent-bit may crash and temporarily stop processing logs, leading to gaps in the audit trail. When this occurs, an error similar to the following is logged.

[error] [input:tail:tail.0] file=/var/log/terraform-enterprise/atlas.log requires a larger buffer size, lines are too long. Skipping file.

Solutions

There are two potential solutions to this issue. The first solution addresses the fluent-bit buffer limit within Terraform Enterprise, while the second addresses a potential system-level cause related to journald rate limiting.

Solution 1: Upgrade Terraform Enterprise

Upgrade your Terraform Enterprise instance to version v202402-1 or newer. In this version, the fluent-bit buffer limit was increased from 32KB to 128KB, which resolves the issue for most log sizes.

If the issue persists after the upgrade, investigate whether any operations, such as a Sentinel run, are producing log outputs that exceed the new 128KB limit.

Solution 2: Adjust journald Rate Limiting

If logs are still missing after upgrading, system-level log rate limiting by journald may be the cause. Check for suppressed message notifications in the journald logs.

$ journalctl | grep -i suppressed

If you see output similar to the following, journald is rate-limiting logs.

printk: systemd: 19 output lines suppressed due to ratelimiting

To disable rate limiting, edit the journald configuration file.

Open the configuration file /etc/systemd/journald.conf in a text editor.
Set the rate limit values to 0 to disable them.
```
RateLimitInterval=0
RateLimitBurst=0
```
Restart the journald service to apply the changes.
```
$ sudo systemctl restart systemd-journald
```

Additional Information

For more details on journald rate limiting, please see this journald rate limiting KB article.