Terraform Enterprise Restart Increase Disk Consumption Over Time by RabbitMQ Disk Allocation

Problem

An observed increased capacity consumption of disk space over time by RabbitMQ on Terraform Enterprise Replicated Deployments running in non-consolidated services mode.

Prerequisites

• Terraform Enterprise Replicated Deployment running in non-consolidated services mode

Cause

• Terraform Enterprise Replicated Deployment, internally runs RabbitMQ to handle messaging.
  This messaging service is run as a docker container under the management of Replicated.
• On each restart of Terraform Enterprise, the tfe-rabbitmq container reserves an allocation of disk space for use by the mnesia database. This is the directory where RabbitMQ node's data is stored. This includes a schema database, message stores, cluster member information and other persistent node state.

On each restart, a new sub-directory is created in /var/lib/rabbitmq/mnesia.

# docker exec --tty --interactive tfe-rabbitmq sh

/ # ls -l /var/lib/rabbitmq/mnesia
total 52
drwxr-xr-x    4 rabbitmq rabbitmq      4096 Apr 24 00:46 rabbit@4d3f11f31e68
-rw-r--r--    1 rabbitmq rabbitmq       148 Apr 24 00:43 rabbit@4d3f11f31e68-feature_flags
drwxr-xr-x    8 rabbitmq rabbitmq      4096 Apr 24 00:43 rabbit@4d3f11f31e68-plugins-expand
-rw-r--r--    1 rabbitmq rabbitmq         3 Apr 24 00:43 rabbit@4d3f11f31e68.pid
drwxr-xr-x    4 rabbitmq rabbitmq      4096 Apr 24 00:41 rabbit@7e2ea827aac8
-rw-r--r--    1 rabbitmq rabbitmq       148 Apr 24 00:29 rabbit@7e2ea827aac8-feature_flags
drwxr-xr-x    8 rabbitmq rabbitmq      4096 Apr 24 00:41 rabbit@7e2ea827aac8-plugins-expand
drwxr-xr-x    4 rabbitmq rabbitmq      4096 Apr 24 00:16 rabbit@d35d8ea4a4c2
-rw-r--r--    1 rabbitmq rabbitmq       148 Apr 24 00:13 rabbit@d35d8ea4a4c2-feature_flags
drwxr-xr-x    8 rabbitmq rabbitmq      4096 Apr 24 00:13 rabbit@d35d8ea4a4c2-plugins-expand
drwxr-xr-x    4 rabbitmq rabbitmq      4096 Apr 24 00:26 rabbit@eb02ea9ca39d
-rw-r--r--    1 rabbitmq rabbitmq       148 Apr 24 00:23 rabbit@eb02ea9ca39d-feature_flags
drwxr-xr-x    8 rabbitmq rabbitmq      4096 Apr 24 00:23 rabbit@eb02ea9ca39d-plugins-expand

Illustration shows four restarts of Terraform Enterprise v202307-1 running without load

Each restart, is reserving a capacity allocation, about 300-500MB each restart.

# docker system df -v | grep rabbitmq
rabbitmq                                                           1         275.4MB

# docker system df -v | grep rabbitmq
rabbitmq                                                           1         564.2MB

# docker system df -v | grep rabbitmq
rabbitmq                                                           1         958.8MB

# docker system df -v | grep rabbitmq
rabbitmq                                                           1         1.401GB

Illustration shows four restarts of Terraform Enterprise v202307-1 running without load

Solutions

• Schedule automated cleanup of the subdirectories from the previous restart.

• Upgrade to Terraform Enterprise v202309-1 or later, where the application containers follow the new Consolidated Services architecture and RabbitMQ is no longer used.

Outcome

Should the increased consumption of disk capacity be of concern - the earlier directories can be safely deleted after each restart of the Terraform Enterprise application.