Tuning ulimit values for Terraform Enterprise – HashiCorp Help Center

Problem

When Terraform Enterprise executes a complex run, its Docker containers may consume resources that exceed operating system limits. This can cause the run to fail with errors related to resource constraints.

Common errors found in the container logs include:

...too many open files
...
...runtime: mlock of signal stack failed
...
...runtime: increase the mlock limit (ulimit -l) or
...
...Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'

Cause

The complexity of a Terraform configuration, including the number of managed resources and providers used, can cause Docker containers to hit resource limits defined by the host operating system, such as the maximum number of open files (nofile) or the maximum locked-in-memory address space (mlock).

Solutions

There are two primary approaches to resolving this issue. We recommend starting with Solution 1 before proceeding to Solution 2.

Solution 1: Refactor the Terraform Configuration

Review the Terraform configuration to determine if its structure contributes to high resource consumption. If possible, refactor the configuration to be more efficient. A well-structured configuration may prevent the resource limit from being reached.

Solution 2: Tune ulimit Values for the Docker Daemon

If refactoring the configuration is not feasible, you can increase the resource limits for the Docker containers by tuning the ulimit values on the Docker daemon.

Note on Container Naming: In Terraform Enterprise versions v202205-1 through v202308-1, container names use the tfe-<service> convention (e.g., tfe-atlas). Older versions may use the ptfe_ prefix (e.g., ptfe_atlas). For more details, refer to the Terraform Enterprise v202205-1 release notes.

Procedure

Identify the problematic container and its current limits. Use the container logs to identify which container is failing and which ulimit needs adjustment (e.g., open files (-n) or max locked memory (-l)). Check the current limits for that container.
```
$ docker exec -it ptfe_redis sh -c "ulimit -a"
```
Update the Docker daemon configuration. Modify the Docker daemon configuration file, typically located at /etc/docker/daemon.json, to set higher default ulimit values. These values will be inherited by all new containers.
```
{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 64000,
      "Soft": 64000
    }
  }
}
```
Restart the Docker daemon. Apply the configuration changes by restarting the Docker service.
```
$ sudo systemctl restart docker
```
Verify the new limits and retry the run. Confirm that the new ulimit values have been applied to the container, then queue the Terraform run again.
```
$ docker container inspect -f "{{.HostConfig.Ulimits}}" <container_name>
```

Additional Information

For more details on Docker daemon settings, refer to the official Docker documentation on the Daemon configuration file.