Tuning ulimit values for Terraform Enterprise

Introduction

When Terraform Enterprise initializes a run, the various Docker containers that make up the application will begin consuming compute resources.

Depending on the complexity of the Terraform configuration (number of managed resources, which providers are used, etc), the Docker containers may hit resource limit constraints defined by the OS and be unable to complete the run.

Problem

When this happens, the following errors are commonly found in the logs:

...too many open files

...runtime: mlock of signal stack failed

...runtime: increase the mlock limit (ulimit -l) or

...Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'

Solution

Note: For installations using Terraform Enterprise v202205-01 through v202308-1, all container names now follow the naming convention of "tfe-<service>" 

Example: 

ptfe_atlas > tfe-atlas
ptfe_archivist > tfe-archivist

Note - older version can have "ptfe" prefix 

More information can be found in the release notes  with a change here.

• First, review the Terraform configuration to determine if the resource limit is caused by an inefficient configuration. If possible, refactor the configuration to prevent the limit issue from happening.
• If updating the Terraform configuration is not an option, consider tuning ulimit values. The process to increase ulimit values are as below:

1. From the log, locate the problematic container and its ulimit values. Typical ulimit types with errors are open files (-n), max locked memory (-l). For example:

    $ docker exec -it ptfe_redis sh -c "ulimit -a"

2. Update Docker daemon configuration with new ulimit values that are higher than current value. The new ulimit value from Docker daemon will be inherited by its containers and may be set in /etc/docker/daemon.json.

    ## Sample `/etc/docker/deamon.json` config
    ...
      "default-ulimits": {
        "nofile": {
          "Hard": 64000,
          "Name": "nofile",
          "Soft": 64000
        }
      },
    ...

3. Restart the Docker daemon:

    $ sudo systemctl restart docker

4. Verify the new ulimit is applied and queue the run again:

   $ docker container inspect -f "{{.HostConfig.Ulimits}}" <container_name>

Additional Information

https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file