Introduction
Problem
An observed increased capacity consumption of disk space over time by RabbitMQ on Terraform Enterprise Replicated Deployments running in non-consolidated services mode.
Prerequisites (if applicable)
- Terraform Enterprise Replicated Deployment running in non-consolidated services mode
Cause
- Terraform Enterprise Replicated Deployment, internally runs RabbitMQ to handle messaging.
This messaging service is run as a docker container under the management of Replicated. - On each restart of Terraform Enterprise, the
tfe-rabbitmqcontainer reserves an allocation of disk space for use by the mnesia. This is the directory where RabbitMQ node's data is stored. This includes a schema database, message stores, cluster member information and other persistent node state.
On each restart, a new sub-directory is created in /var/lib/rabbitmq/mnesia.
# docker exec --tty --interactive tfe-rabbitmq sh
/ # ls -l /var/lib/rabbitmq/mnesia
total 52
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:46 rabbit@4d3f11f31e68
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:43 rabbit@4d3f11f31e68-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:43 rabbit@4d3f11f31e68-plugins-expand
-rw-r--r-- 1 rabbitmq rabbitmq 3 Apr 24 00:43 rabbit@4d3f11f31e68.pid
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:41 rabbit@7e2ea827aac8
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:29 rabbit@7e2ea827aac8-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:41 rabbit@7e2ea827aac8-plugins-expand
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:16 rabbit@d35d8ea4a4c2
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:13 rabbit@d35d8ea4a4c2-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:13 rabbit@d35d8ea4a4c2-plugins-expand
drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:26 rabbit@eb02ea9ca39d
-rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:23 rabbit@eb02ea9ca39d-feature_flags
drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:23 rabbit@eb02ea9ca39d-plugins-expand
Illustration shows four restarts of Terraform Enterprise v202307-1 running without load
Each restart, is reserving a capacity allocation, about 300-500MB each restart.
# docker system df -v | grep rabbitmq
rabbitmq 1 275.4MB
# docker system df -v | grep rabbitmq
rabbitmq 1 564.2MB
# docker system df -v | grep rabbitmq
rabbitmq 1 958.8MB
# docker system df -v | grep rabbitmq
rabbitmq 1 1.401GB
Illustration shows four restarts of Terraform Enterprise v202307-1 running without load
Overview of possible solutions (if applicable)
Solutions:
-
Schedule automated cleanup of the subdirectories from the previous restart.
-
Upgrade to Terraform Enterprise v202309-1 or later, where the application containers follow the new Consolidated Services architecture and RabbitMQ is no longer used.
Outcome
Should the increased consumption of disk capacity be of concern - the earlier directories can be safely deleted after each restart of the Terraform Enterprise application.
Additional Information
If you encounter issues after trying the solution above, please reach out to HashiCorp Support for assistance.