Resolving 'No Space Left on Device' and 'Out of Shared Memory' Postgres Errors in Terraform Enterprise Mounted Disk (FDO)

Problem

Terraform Enterprise is experiencing PostgreSQL performance degradation accompanied by shared memory errors:

# /var/log/terraform-enterprise/postgres.log
ERROR:  out of shared memory
ERROR:  could not resize shared memory segment "/PostgreSQL.2143585788" to 33554432 bytes: No space left on device  

Prerequisites 

• Terraform Enterprise FDO
• Operational Mode: Mounted Disk

Cause

• ERROR:  out of shared memory - Root cause identified as sub-optimal settings for these PostgreSQL parameters in a High Load environment.

shared_buffers = 128MB
max_locks_per_transaction = 64

• ERROR:  could not resize shared memory segment - Root cause identified as sub-optimal setting for the inter-process Shared Memory in a High Load environment.

# Default Docker's Shared Memory Size is 64MB (/dev/shm)
dynamic_shared_memory_type = posix

Solution:

1. Increase the values for shared_buffers and max_locks_per_transaction in PostgreSQL

• Copy the postgresql.conf from the TFE container to the Host's Mounted Disk path, e.g.

  docker cp \
    terraform-enterprise-tfe-1:/etc/postgresql/postgresql.conf \
    $(docker inspect terraform-enterprise-tfe-1 --format='{{index .Config.Labels "com.docker.compose.project.working_dir"}}')/postgresql.conf

• Edit the postgresql.conf.

  vim $(docker inspect terraform-enterprise-tfe-1 --format='{{index .Config.Labels "com.docker.compose.project.working_dir"}}')/postgresql.conf

• Add or Modify the following parameters.

  shared_buffers = 512MB
  max_locks_per_transaction = 512

2. Edit the Docker Compose YAML file: 

• Set the Docker Shared Memory 

  ---
  name: terraform-enterprise
  services:
    tfe:
      image: images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#>
      shm_size: '1gb'   <-- Sets /dev/shm size
      environment:
  ... truncated   

• Create a Bind Mount to mount the modified postgresql.conf inside the Terraform Enterprise container.

  volumes:
  ... truncated
    - type: bind
      source: /<mounted_disk_path_on_host>/postgresql.conf
      target: /etc/postgresql/postgresql.conf   

3. Apply the changes

• Restart the container to apply the new configuration changes

  # docker compose up -d

• Verify the new shared memory value in Docker

  # docker compose exec tfe df -h|grep shm

4. Check new Postgres parameter values

• Launch a terminal inside the Terraform Enterprise container

  # docker compose exec tfe bash

• Connect to Postgres using the CLI

  # psql $(tfectl app config --full|jq -r '."database"."url"')

• Display the new paramater values

  show shared_buffers; show max_locks_per_transaction;

Outcome

Monitor the Postgres log for ERROR messages to confirm the issue has been mitigated.

# docker compose exec tfe sh -c \
"tail -f /var/log/terraform-enterprise/postgres.log \
| grep --line-buffered 'ERROR'"

Additional Information

• If the issue is not resolved please Open a New Support Request.