Problem
Allocation is unable to place due to Resource exhausted. Dimension disk exhausted on x node
Here is an example of what it may look like:
Failed to place 1 allocation(s):
* Job "example-job" (Task Group "web" count: 1)
* No nodes are available to place allocation:
- Dimension "disk" exhausted on 1 nodesCause
alloc/data partition or the /var directory, which is typically used for the ephemeral disk by default.
Solutions:
ephemeral_disk stanza in your job configuration. Keep in mind that all tasks within the same group share the same ephemeral disk space. It is not recommended to reduce the allocation below 300MB, as this space is essential for data migration and logging. Additionally, dead allocations that have not yet been garbage-collected will count against the 300MB reservation.
To address the "Dimension disk exhausted" error, the recommended solution is to free up disk space on the /var partition of the Nomad client host.
If freeing up sufficient disk space is not feasible, you may consider changing the default location of the alloc_dir in the client configuration stanza. However, this approach has potential implications for task execution speed. If the new alloc_dir is not on the same disk as usr and etc directories used in the chroot environment, Nomad will need to copy files instead of creating symlinks, which can slow down task building. For this reason, we strongly recommend thoroughly testing this configuration change based on the type of underlying storage hardware to assess its impact.
Additional Information
The ephemeral_disk stanza describes the ephemeral disk requirements of the group. Ephemeral disks can be marked as sticky and support live data migration. Additional references below: