How to Identify when Terraform Enterprise runs exceed memory capacity

Problem

During a Terraform Enterprise run, the plan or apply operation may fail with one of the following errors:

• Killed
• connection is shut down
• unexpected EOF
• Error: rpc error: code = Canceled desc = context canceled

Cause

These errors usually indicate that the Terraform configuration consumed more memory than was available in the agent's container, causing the operating system's out-of-memory (OOM) killer to terminate the process.

Solution

To confirm if an OOM event caused the failure, you can inspect the system logs on the Terraform Enterprise host for the timeframe of the failed run.

1. In the Terraform Enterprise UI, note the start and end time of the failed run.

2. Connect to the Terraform Enterprise host and use the journalctl command to search for OOM kill events within that timeframe. Replace the example timestamps with the ones from your failed run.

   $ journalctl --since "2023-06-05 11:10:10" --until "2023-06-05 12:05:50" | grep "oom-kill"

3. Analyze the command output. If OOM events occurred, the output will resemble the following, confirming that a process was terminated due to excessive memory usage.

   [ 3012.565547] terraform-provi invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
   [ 3041.315832] terraform invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
   [ 3042.204040] terraform invoked oom-killer: gfp_mask=0x50, order=0, oom_score_adj=0

If the command produces no output, the errors are likely not caused by an OOM condition.

Additional Information

• Troubleshoot Plan operation failures in Terraform Enterprise
• Memory and Concurrency settings in Terraform Enterprise