Problem
An observed increased capacity consumption of disk space over time by RabbitMQ on Terraform Enterprise Replicated Deployments running in non-consolidated services mode.
Prerequisites
- Terraform Enterprise Replicated Deployment running in non-consolidated services mode
Cause
- Terraform Enterprise Replicated Deployment, internally runs RabbitMQ to handle messaging.
This messaging service is run as a docker container under the management of Replicated. - On each restart of Terraform Enterprise, the
tfe-rabbitmq
container reserves an allocation of disk space for use by the mnesia database. This is the directory where RabbitMQ node's data is stored. This includes a schema database, message stores, cluster member information and other persistent node state.
On each restart, a new sub-directory is created in /var/lib/rabbitmq/mnesia
.
# docker exec --tty --interactive tfe-rabbitmq sh
/ # ls -l /var/lib/rabbitmq/mnesia
total 52
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:46 rabbit@4d3f11f31e68
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:43 rabbit@4d3f11f31e68-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:43 rabbit@4d3f11f31e68-plugins-expand
-rw-r--r-- 1 rabbitmq rabbitmq 3 Apr 24 00:43 rabbit@4d3f11f31e68.pid
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:41 rabbit@7e2ea827aac8
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:29 rabbit@7e2ea827aac8-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:41 rabbit@7e2ea827aac8-plugins-expand
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:16 rabbit@d35d8ea4a4c2
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:13 rabbit@d35d8ea4a4c2-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:13 rabbit@d35d8ea4a4c2-plugins-expand
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:26 rabbit@eb02ea9ca39d
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:23 rabbit@eb02ea9ca39d-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:23 rabbit@eb02ea9ca39d-plugins-expand
Illustration shows four restarts of Terraform Enterprise v202307-1 running without load
Each restart, is reserving a capacity allocation, about 300-500MB each restart.
# docker system df -v | grep rabbitmq
rabbitmq 1 275.4MB
# docker system df -v | grep rabbitmq
rabbitmq 1 564.2MB
# docker system df -v | grep rabbitmq
rabbitmq 1 958.8MB
# docker system df -v | grep rabbitmq
rabbitmq 1 1.401GB
Illustration shows four restarts of Terraform Enterprise v202307-1 running without load
Solutions
-
Schedule automated cleanup of the subdirectories from the previous restart.
-
Upgrade to Terraform Enterprise v202309-1 or later, where the application containers follow the new Consolidated Services architecture and RabbitMQ is no longer used.
Outcome
Should the increased consumption of disk capacity be of concern - the earlier directories can be safely deleted after each restart of the Terraform Enterprise application.