Introduction

Scenarios

• Regular Nomad host maintenance requires the host operating system to reboot.
• Nomad host outage requires the hosts to be started back up.

Recommendation

Important

• First start with the Nomad server cluster hosts, then after that is done, move on to the Nomad client nodes.
• Staggering the server reboots is recommended. It is best to use the Nomad Upgrade Process as a template.
• Take a Nomad snapshot before any activity that may cause an outage.

OS Reboot on Nomad Server Cluster Hosts

Do one Nomad server host on the cluster at a time.

1. Stop Nomad gracefully (sudo systemctl stop nomad).
2. If the server was the leader, wait for a follower to gain leadership.
3. Verify the cluster health. (nomad server members).
4. Reboot the server host.
5. Start Nomad (sudo systemctl start nomad).
6. Verify the cluster health.

OS Reboot on Nomad Client Node Hosts

• You can do one client node at a time or multiple client nodes at a time.  Follow their respective steps.
• When rebooting the Nomad Client Node, it is best to drain the node so all of its allocations are migrated to other nodes to avoid disruption of Nomad job tasks.
• All Nomad jobs have parameters set with defaults, same for the Migrate Stanza:
  • max_parallel (int: 1)
  • health_check (string: "checks"
  • min_healthy_time (string: "10s")
  • healthy_deadline (string: "5m")

One Client Node Host at a Time

1. Drain the client node (nomad node drain -enable -yes <Node ID>).
2. Check the client node status (nomad node status <Node ID>).
3. Stop Nomad gracefully (sudo systemctl stop nomad).
4. Reboot the client node's host OS.
5. Check the client node status.
6. Restore client node eligibility (nomad node drain -disable -yes <Node ID>).
   1. If you get "Error toggling drain mode: Unexpected response code: 500 (no servers)", run the command from a Nomad Server host.
7. Check the client node status.

Multiple Client Node Hosts at a Time

1. See Workload Migration, Drain multiple nodes for details.  It includes shell scripting that can help automate manipulating multiple nodes at once.
2. Check the node status (nomad node status -allocs).
3. Set client nodes ineligible (nomad node eligibility -disable <Node ID>).
4. Check the node status
5. Drain client nodes (nomad node drain -enable -yes <Node ID>).
6. Check the node status. 
7. Stop Nomad gracefully (sudo systemctl stop nomad).
8. Reboot the drained client nodes' hosts OS.
9. Check the node status.
10. Restore nodes' eligibility (nomad node drain -disable -yes <Node ID>).
    1. If you get "Error toggling drain mode: Unexpected response code: 500 (no servers)", run the command from a Nomad Server host.
11. Check the node status.

Additional Information

• Advanced Node Draining in HashiCorp Nomad
• Nomad Command: node drain
• Nomad Command: node eligibility
• Nomad Command: operator snapshot restore
• Nomad Command: operator snapshot save
• Nomad Upgrade Process
• Nomad Workload Migration