Problem
Disk space consumption by RabbitMQ increases over time after each restart of a Terraform Enterprise Replicated deployment running in non-consolidated services mode.
Prerequisites
- A Terraform Enterprise Replicated deployment running in non-consolidated services mode.
Cause
Terraform Enterprise Replicated deployments use RabbitMQ internally to handle messaging. This messaging service runs as a Docker container for the RabbitMQ message queue under the management of Replicated.
On each restart of Terraform Enterprise, the tfe-rabbitmq container reserves an allocation of disk space for its mnesia database. This directory stores the RabbitMQ node's data, including the schema database, message stores, and other persistent node state. A new subdirectory is created in /var/lib/rabbitmq/mnesia with each restart.
To view these directories, execute a shell in the tfe-rabbitmq container and list the contents of the mnesia directory.
# docker exec --tty --interactive tfe-rabbitmq sh / # ls -l /var/lib/rabbitmq/mnesia total 52 drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:46 rabbit@4d3f11f31e68 -rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:43 rabbit@4d3f11f31e68-feature_flags drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:43 rabbit@4d3f11f31e68-plugins-expand -rw-r--r-- 1 rabbitmq rabbitmq 3 Apr 24 00:43 rabbit@4d3f11f31e68.pid drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:41 rabbit@7e2ea827aac8 -rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:29 rabbit@7e2ea827aac8-feature_flags drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:41 rabbit@7e2ea827aac8-plugins-expand drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:16 rabbit@d35d8ea4a4c2 -rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:13 rabbit@d35d8ea4a4c2-feature_flags drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:13 rabbit@d35d8ea4a4c2-plugins-expand drwxr-xr-x 4 rabbitmq rabbitmq 4096 Apr 24 00:26 rabbit@eb02ea9ca39d -rw-r--r-- 1 rabbitmq rabbitmq 148 Apr 24 00:23 rabbit@eb02ea9ca39d-feature_flags drwxr-xr-x 8 rabbitmq rabbitmq 4096 Apr 24 00:23 rabbit@eb02ea9ca39d-plugins-expand
Each restart reserves a capacity allocation of approximately 300-500MB. The following output illustrates the cumulative disk usage after four restarts.
# docker system df -v | grep rabbitmq rabbitmq 1 275.4MB # docker system df -v | grep rabbitmq rabbitmq 1 564.2MB # docker system df -v | grep rabbitmq rabbitmq 1 958.8MB # docker system df -v | grep rabbitmq rabbitmq 1 1.401GB
Solutions
There are two approaches to resolve this issue.
Solution 1: Manually Clean Up Directories
You can schedule an automated cleanup of the older subdirectories created by previous restarts. The directories from previous sessions can be safely deleted after each restart of the Terraform Enterprise application.
Solution 2: Upgrade to a Later Version
Upgrade to Terraform Enterprise v202309-1 or later. In these versions, the application containers follow the new Consolidated Services architecture, and RabbitMQ is no longer used.
Outcome
By implementing one of the solutions, you can prevent the uncontrolled growth of disk consumption caused by RabbitMQ data directories on Terraform Enterprise restarts.