Troubleshoot Plan operation failures in Terraform Enterprise

Introduction

When a run starts in Terraform Enterprise, it spawns a container from a default or custom worker image or custom agent image. The system then executes a series of commands inside the container, including Terraform CLI operations like terraform plan.

This article describes potential failures that can occur during the plan operation and provides methods to resolve them.

Problem

A Terraform plan operation may fail for several reasons, each producing a distinct error message.

• Container Out of Memory: The plan demands more memory than the worker container is allocated. The run output may end with Killed, or contain messages like connection is shut down, unexpected EOF, or Error: rpc error: code = Canceled desc = context canceled.
• Invalid Symlinks: The configuration contains symbolic links to files or directories outside of the source code repository, which Terraform Enterprise prohibits for security reasons. The plan fails with the error: Setup failed: failed unpacking terraform config: Invalid symlink ("<<SOURCE>> -> <<DESTINATION>>") has absolute target.

• Plan Timeout: The plan operation exceeds the configured time limit, which is two hours by default. The run output displays a system message indicating a timeout.

  ------------ Terraform Enterprise System Message ------------
  WARNING: This plan has timed out and will now terminate!
  
  Terraform Enterprise enforces a 30m0s maximum run time for this operation. Please
  review the logs above to determine why the run has exceeded its timeout. You
  can re-run this operation by queueing a new plan in Terraform Enterprise.
  -------------------------------------------------------------

Solutions

Solution 1: Address "Out of Memory" Errors

1. Upgrade Terraform CLI: Terraform CLI versions 0.12.29 and later include changes that can significantly decrease memory usage. We recommend upgrading to the latest version if possible.
2. Increase Worker Memory: If upgrading is not an option or does not resolve the issue, increase the memory capacity of the worker container. You can adjust the Maximum memory per run setting in the installer dashboard at https://TFE_HOSTNAME:8800 under Settings > Capacity. You may need to incrementally increase this value to find the right allocation.
3. Adjust Host Memory: When you increase the maximum memory per run, the total memory required for the host machine also increases. Ensure the host machine's memory capacity is adjusted according to the memory adjustment guidelines. For more details, refer to the Capacity and Performance documentation.

Solution 2: Resolve Invalid Symlink Errors

To resolve this security-related failure, you must remove all symlinks that point to absolute paths or locations outside of the configuration's root directory from your source code repository.

Solution 3: Mitigate Plan Timeouts

1. Limit Resources per Workspace: A large number of resources can increase plan time due to provider API communication. Limiting the number of resources in a single workspace can help reduce bottlenecks.
2. Increase Plan Timeout: If you cannot optimize the configuration further, you can increase the plan timeout. Navigate to the Site Admin settings by selecting your user profile icon and choosing Site Admin, or by navigating directly to https://TFE_HOSTNAME/app/admin/settings.
3. Increase Parallel Operations: By default, Terraform runs with 10 parallel operations. You can increase this by setting the TFE_PARALLELISM environment variable on the workspace, as described in the special environment variables documentation.
4. Upgrade Host Instance CPU: A higher CPU count on the host instance allows Terraform Enterprise to process the configuration faster. Note that external factors, such as provider API response times, may still limit performance.
5. Upgrade Disk IOPS: With larger configurations or high concurrency, Terraform Enterprise performance can be limited by disk I/O. In cloud environments, you may need to increase the allocated IOPS for the disk.
6. Refactor Configuration: Analyze and refactor your Terraform configuration to reduce the complexity of the resource graph, which can shorten processing time.

Additional Information

It is important to understand the potential consequences of these changes. For example, increasing the memory allocated to each run may exhaust system memory if the host machine's capacity is not also increased. Always consider the system-wide impact of any adjustments.