Removing Orphaned Mounts from Nomad Allocation Directory

Applies To

• Nomad versions prior to v0.6.1 on normal and unexpected exits
• All Nomad versions on unexpected exits

Issue

In versions of Nomad prior to v0.6.1, When stopping Nomad, some mounted read-only filesystems might remain:

• «nomad_data_dir»/alloc/«task-alloc-id»/«task-name»/secrets
• «nomad_data_dir»/alloc/«task-alloc-id»/«task-name»/dev
• «nomad_data_dir»/alloc/«task-alloc-id»/«task-name»/proc
• «nomad_data_dir»/alloc/«task-alloc-id»/«task-name»/alloc

Attempting to delete them can result in messages like:

rm: cannot remove ‘«nomad_data_dir»/alloc/«task-alloc-id»/«task-name»/secrets’: Device or resource busy

or

rm: cannot remove ‘«nomad_data_dir»/alloc/«task-alloc-id»/«task-name»/proc/...’: Read-only file system

Resolution

These folders remain available by design. Nomad uses a garbage collection process to destroy these folders, this allows an operator to inspect the state of an allocation that has yet to be garbage collected even if the operator chooses to stop the Nomad process. Ordinarily, these mounts are cleaned up by Nomad’s garbage collection process and would not cause any issues.

There are two options to remove these directories beyond Nomad’s internally scheduled garbage collection.

• Stimulate a garbage collection run using the system/gc API endpoint
• Unmount the folders manually

Stimulate Garbage Collection

The HTTP API’s system/gc endpoint can be used to tell Nomad to immediately make a garbage collection pass. In the case of a completely drained client node, this will serve to remove any remaining allocation state data and would prevent encountering the read-only tmpfs mounts.

curl -XPUT http://127.0.0.1:4646/v1/system/gc

This process does require that the Nomad client process be up and available and that the nodes are drained of all running allocations. Draining the node is necessary to ensure that all of the allocations are stopped and eligible for garbage collation when run manually.

Note: Stimulating garbage collection via the gc endpoint will perform garbage collection on every node in the cluster.

Manual Unmounting

Since these read-only file systems are regular Linux mounts, you can use the umount command to unmount them. This could be done in a one-liner similar to the following:

export NOMAD_DATA_ROOT=«Path to your Nomad data_dir»

for ALLOC in `ls -d $NOMAD_DATA_ROOT/alloc/*`; do for JOB in `ls ${ALLOC}| grep -v alloc`; do umount ${ALLOC}/${JOB}/secrets; umount ${ALLOC}/${JOB}/dev; umount ${ALLOC}/${JOB}/proc; umount ${ALLOC}/${JOB}/alloc; done; done

Once the directories are unmounted, all of the remaining contents can be deleted.