Running Nomad Autoscaler Directly on a Host – HashiCorp Help Center

Introduction

Nomad Autoscaler is a powerful daemon designed to provide autoscaling capabilities for Nomad clusters. It supports Horizontal Application Autoscaling, Horizontal Cluster Autoscaling, and Dynamic Application Sizing (Enterprise). While comprehensive documentation and tutorials are available for running Nomad Autoscaler as a containerized task managed by Nomad, this article focuses on scenarios where deploying it directly on the host machine is preferred or necessary.

Note: This article specifically covers "Horizontal Application Autoscaling," which involves adding or removing Nomad allocations to adjust application resources. The concepts presented here can be adapted for "Dynamic Application Sizing (Enterprise)," which facilitates vertical scaling.

Expected Outcome

By following the instructions and examples provided in this article, you will be able to successfully configure and run the Nomad Autoscaler daemon directly on a host machine, demonstrating its functionality with the provided examples.

Use Case

This approach to running Nomad Autoscaler is particularly useful in environments where:

Containerization of the autoscaler itself is not feasible or desired.
Direct host access and control over the autoscaler process are required.
Existing infrastructure or security policies limit the deployment of the autoscaler as a Nomad job.

Prerequisites

Before proceeding, ensure you have the following in place:

A configured Nomad environment: Set up a Nomad cluster and a Prometheus monitoring system. We recommend using the tutorials "Scale your Nomad cluster with horizontal cluster autoscaling" (for cloud environments) or "Scale an application with the Nomad Autoscaler" (for VM environments) as a baseline for your environment setup.
Nomad Autoscaler binary: Download and install the Nomad Autoscaler binary on the host where you intend to run it. Refer to the official Nomad Autoscaler documentation for installation instructions.
Existing Nomad Autoscaler job stopped (if applicable): If you previously had Nomad Autoscaler running as a Nomad job (e.g., aws_autoscaler.nomad), ensure that job is stopped before attempting to run it directly on the host to avoid conflicts.

Procedure

The following steps guide you through configuring and running the Nomad Autoscaler directly from the host. The examples provided assume an environment configured using the "Scale your Nomad cluster with horizontal cluster autoscaling" tutorial.

Create Configuration Directories

Create dedicated directories on your host to store the Nomad Autoscaler configuration and policy files. For example:

sudo mkdir -p /etc/nomad-autoscaler-config.d
sudo mkdir -p /etc/nomad-autoscaler-policy.d

Create the Autoscaler Configuration File

Create a file named autoscaler-config.hcl and place it in the /etc/nomad-autoscaler-config.d directory. This file defines the general settings for the Nomad Autoscaler.

Example autoscaler-config.hcl:

log_level = "TRACE"

nomad {
  address = "http://172.31.18.13:4646"  # Replace with your Nomad client address
}

apm "prometheus" {
  driver = "prometheus"
  config = {
    address = "http://172.31.18.13:25165" # Replace with your Prometheus address
  }
}

target "aws-asg" {
  driver = "aws-asg"
  config = {
    aws_region = "us-east-1" # Replace with your AWS region
  }
}

strategy "target-value" {
  driver = "target-value"
}

Important:

Replace http://172.31.18.13:4646 with the actual address of your Nomad client.
Replace http://172.31.18.13:25165 with the actual address of your Prometheus server.
Replace us-east-1 with your AWS region if using AWS Auto Scaling Groups.

Create the Autoscaler Policy File

Create a file named autoscaler-policy.hcl and place it in the /etc/nomad-autoscaler-policy.d directory. This file defines the scaling policies, including metrics to monitor and actions to take.

Example autoscaler-policy.hcl:

scaling "aws_cluster_policy" {
  enabled = true
  min     = 1
  max     = 2

  policy {
    cooldown            = "2m"
    evaluation_interval = "1m"

    check "cpu_allocated_percentage" {
      source = "prometheus"
      query  = "sum(nomad_client_allocated_cpu{node_class=\"hashistack\"}*100/(nomad_client_unallocated_cpu{node_class=\"hashistack\"}+nomad_client_allocated_cpu{node_class=\"hashistack\"}))/count(nomad_client_allocated_cpu{node_class=\"hashistack\"})"

      strategy "target-value" {
        target = 70
      }
    }

    check "mem_allocated_percentage" {
      source = "prometheus"
      query  = "sum(nomad_client_allocated_memory{node_class=\"hashistack\"}*100/(nomad_client_unallocated_memory{node_class=\"hashistack\"}+nomad_client_allocated_memory{node_class=\"hashistack\"}))/count(nomad_client_allocated_memory{node_class=\"hashistack\"})"

      strategy "target-value" {
        target = 70
      }
    }

    target "aws-asg" {
      dry-run             = "false"
      aws_asg_name        = "main-mhc-mws-nomad_client" # Replace with your AWS ASG name
      node_class          = "hashistack"
      node_drain_deadline = "5m"
   }
  }
}

Important:

Replace main-mhc-mws-nomad_client with the actual name of your AWS Auto Scaling Group.
Adjust min and max values to reflect your desired minimum and maximum cluster sizes.
Modify the query parameters for CPU and memory checks if your Prometheus metrics or node classes differ.

Create a Script to Run the Nomad Autoscaler Agent

Create a bash script, for example, run-autoscaler.sh, to execute the Nomad Autoscaler agent. This script will configure the agent to use the configuration and policy files you created.

Example run-autoscaler.sh:

#!/bin/bash

# Ensure the Nomad Autoscaler binary is in your PATH or provide the full path
# Example: /usr/local/bin/nomad-autoscaler

nomad-autoscaler agent \
  -config=/etc/nomad-autoscaler-config.d \
  -policy-dir=/etc/nomad-autoscaler-policy.d \
  -enable-debug \
  2>&1 | tee "nomad-autoscaler_$(date -u -Is).log"

Make the script executable:

chmod +x run-autoscaler.sh

Now, you can run the Nomad Autoscaler by executing this script:

./run-autoscaler.sh

The output will be logged to a file named nomad-autoscaler_YYYY-MM-DDTHH:MM:SSZ.log (e.g., nomad-autoscaler_2025-07-28T16:34:17Z.log).

Initiate Autoscaling

To observe and validate the autoscaling behavior, follow the procedures outlined in these tutorials:

For Application Autoscaling: "Scale an application with the Nomad Autoscaler, Review and run the webapp job."
For Cluster Autoscaling: "Scale your Nomad cluster with horizontal cluster autoscaling, Generate application load."

By generating load or adjusting your application's resource demands, you should see the Nomad Autoscaler respond by adding or removing Nomad allocations or cluster nodes based on the policies defined in your autoscaler-policy.hcl file.

Additional Information

Nomad Autoscaler References