This article provides a comprehensive explanation for scenarios where repetitive audit logs appear under a specific Terraform Cloud (TFC) Run ID, clarifies the role of health assessments in this behaviour, and offers a solution to resolve discrepancies related to Vault access roles.
Problem Overview
Repetitive audit logs tied to a specific TFC Run ID being logged daily. Normally, a Run ID is unique for each execution and cannot be reused. Despite this, the same Run ID appears repeatedly in the logs. Additionally, the logs reference an old Vault role (tfc-global-identity-old) even though the TFC Vault Access Role environment variable (TFC_VAULT_RUN_ROLE) is updated to a new role (tfc-global-identity-new).
Root Cause Analysis
If the health assessment feature is enabled in a TFC organization, it relies on a snapshot of the workspace's environment from the last successfully applied run. Health assessments run periodically (typically every 24 hours) using a snapshot of the previous environment until a new run is applied. If the TFC_VAULT_RUN_ROLE environment variable is updated but not applied via a new run, the health checks continue to use the old snapshot, resulting in logs referencing the outdated Vault role.
Behaviour of Health Assessments:
- They utilize the last applied snapshot for analysis.
- Repetitive logs with the same Run ID are generated as part of periodic health checks.
- A new Run ID in logs indicates a regular run was initiated and applied in the workspace.
Solution
To address the issue:
-
Initiate and Apply a New Run: Trigger a new run in the affected workspace to update the environment snapshot with the new
TFC_VAULT_RUN_ROLEvalue. -
Verify Log Behavior: Confirm that new audit logs reference the updated Vault role (
tfc-global-identity-new).
Conclusion
This behaviour is expected when updates to critical environment variables are not applied through a new run. By initiating and applying a new run, the environment snapshot is refreshed, resolving discrepancies in audit logs.