Problem
Allocation is unable to place due to Resource exhausted. Dimension disk exhausted on x node
Cause
The likely cause of this is due to lack of disk space in the /var partition of the host
Solutions:
Keep in mind that Nomad reserves by default 300MB for ephemeral disk for each job placement. You can increase the sizing by using the ephemeral_disk stanza in your job. All tasks in the in the group will share the same ephemeral disk. It is not recommended to go below 300MB as it is used for data migration as well as logging. It should also be noted that dead allocations that have not been GC'd will also counts against the 300MB reservation.
The recommend way to resolve the Dimension disk exhausted error is to free up disk space on the /var partition of the Nomad client host.
Alternatively if you are unable to free up enough disk space to place the allocation, you may also change the default location of the alloc_dir in the client configuration stanza. Unfortunately this could have potential impact on the execution speed of task building if chroot is not on the same disk as usr and etc. Since the alloc_dir isn't on the same folder as the files Nomad puts in the chroot, it has to copy them rather than making symlinks. For this reason we highly recommend thorough testing this depending on the type of the underlying storage hardware.
Additional Information
The ephemeral_disk stanza describes the ephemeral disk requirements of the group. Ephemeral disks can be marked as sticky and support live data migration. Additional references below: