Problem
Terraform Enterprise is experiencing PostgreSQL performance degradation accompanied by shared memory errors:
# /var/log/terraform-enterprise/postgres.log
ERROR: out of shared memory
ERROR: could not resize shared memory segment "/PostgreSQL.2143585788" to 33554432 bytes: No space left on device Prerequisites
- Terraform Enterprise FDO
- Operational Mode: Mounted Disk
Cause
-
ERROR: out of shared memory- Root cause identified as sub-optimal settings for these PostgreSQL parameters in a High Load environment.
shared_buffers = 128MB
max_locks_per_transaction = 64
-
ERROR: could not resize shared memory segment- Root cause identified as sub-optimal setting for the inter-process Shared Memory in a High Load environment.
# Default Docker's Shared Memory Size is 64MB (/dev/shm)
dynamic_shared_memory_type = posix
Solution:
1. Increase the values for shared_buffers and max_locks_per_transaction in PostgreSQL
-
Copy the postgresql.conf from the TFE container to the Host's Mounted Disk path, e.g.
docker cp \ terraform-enterprise-tfe-1:/etc/postgresql/postgresql.conf \ $(docker inspect terraform-enterprise-tfe-1 --format='{{index .Config.Labels "com.docker.compose.project.working_dir"}}')/postgresql.conf -
Edit the postgresql.conf.
vim $(docker inspect terraform-enterprise-tfe-1 --format='{{index .Config.Labels "com.docker.compose.project.working_dir"}}')/postgresql.conf -
Add or Modify the following parameters.
shared_buffers = 512MB max_locks_per_transaction = 512
2. Edit the Docker Compose YAML file:
-
Set the Docker Shared Memory
--- name: terraform-enterprise services: tfe: image: images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#> shm_size: '1gb' <-- Sets /dev/shm size environment: ... truncated -
Create a Bind Mount to mount the modified postgresql.conf inside the Terraform Enterprise container.
volumes: ... truncated - type: bind source: /<mounted_disk_path_on_host>/postgresql.conf target: /etc/postgresql/postgresql.conf
3. Apply the changes
-
Restart the container to apply the new configuration changes
# docker compose up -d -
Verify the new shared memory value in Docker
# docker compose exec tfe df -h|grep shm
4. Check new Postgres parameter values
-
Launch a terminal inside the Terraform Enterprise container
# docker compose exec tfe bash -
Connect to Postgres using the CLI
# psql $(tfectl app config --full|jq -r '."database"."url"') -
Display the new paramater values
show shared_buffers; show max_locks_per_transaction;
Outcome
Monitor the Postgres log for ERROR messages to confirm the issue has been mitigated.
# docker compose exec tfe sh -c \
"tail -f /var/log/terraform-enterprise/postgres.log \
| grep --line-buffered 'ERROR'"Additional Information
If the issue is not resolved please Open a New Support Request.