Nomad Autoscaler - Using Multiple APMs for Horizontal Autoscaling and Dynamic Application Sizing (DAS)

Avatar
Ron Savoia
  • Updated

Introduction

Nomad Autoscaler is a horizontal application and cluster autoscaler for Nomad. Additionally, the enterprise version of Nomad Autoscaler supports Dynamic Application Sizing (DAS). While Nomad Autoscaler and Nomad work hand in hand, the Nomad Autoscaler is built and released separately to Nomad.  Nomad Autoscaler supports several APMs for horizontal scaling; however, currently, Prometheus must be the APM used for Dynamic Application Sizing (DAS).  Many companies, especially those using AWS, use Datadog as the default APM, so we will focus on Datadog and Prometheus.  This article covers how one can configure Nomad Autoscaler to utilize two APMs for horizontal scaling and DAS.

 

Prerequisites (if applicable)

  • Nomad & Consul cluster currently running (Nomad Enterprise (with the Multi-Cluster & Efficiency module) is needed for DAS)
  • Datadog agent installed on each Nomad node

Overview

At a high-level the setup process includes:

  • Adding telemetry configuration for nomad
  • Deploy Prometheus
  • Deploy Nomad Autoscaler (Nomad Autoscaler Enterprise is needed for DAS)
  • Deploy load test apps for validation testing

This article utilizes Nomad to deploy both Prometheus and the Nomad Autoscaler as apps within the cluster.  While this is not a hard requirement, it is recommended to deploy Nomad Autoscaler as an app within the cluster. 

Procedures

  • Add telemetry configuration for nomad
    -  On each of the nomad nodes to be monitored, add the following telemetry stanza to your nomad.hcl file. 
    telemetry {
    publish_allocation_metrics = true
    publish_node_metrics = true
    datadog_address = "localhost:8125"
    collection_interval = "10s"
    prometheus_metrics = true
    }
    - Restart Nomad for the change to take effect.

  • Deploy Prometheus
    -  Deploy the Prometheus job below

    --- prometheus.nomad ---

    job "prometheus" {
    datacenters = ["dc1"]

    group "prometheus" {
    count = 1

    network {
    port "prometheus_ui" {
    static = 9090
    to = 9090
    }
    }

    task "prometheus" {
    driver = "docker"

    config {
    image = "prom/prometheus:v2.25.0"
    ports = ["prometheus_ui"]

    args = [
    "--config.file=/etc/prometheus/config/prometheus.yml",
    "--storage.tsdb.path=/prometheus",
    "--web.console.libraries=/usr/share/prometheus/console_libraries",
    "--web.console.templates=/usr/share/prometheus/consoles",
    ]

    volumes = [
    "local/config:/etc/prometheus/config",
    ]
    }

    template {
    data = <<EOH
    ---
    global:
    scrape_interval: 1s
    evaluation_interval: 1s

    scrape_configs:
    - job_name: nomad
    metrics_path: /v1/metrics
    params:
    format: ['prometheus']
    static_configs:
    - targets: ['{{ env "attr.unique.network.ip-address" }}:4646']

    - job_name: consul
    metrics_path: /v1/agent/metrics
    params:
    format: ['prometheus']
    static_configs:
    - targets: ['{{ env "attr.unique.network.ip-address" }}:8500']
    EOH

    change_mode = "signal"
    change_signal = "SIGHUP"
    destination = "local/config/prometheus.yml"
    }

    resources {
    cpu = 500
    memory = 512
    }

    service {
    name = "prometheus"
    port = "prometheus_ui"

    check {
    type = "http"
    path = "/-/healthy"
    interval = "10s"
    timeout = "2s"
    }
    }
    }
    }
    }


  • Deploy Nomad Autoscaler
    -  Update the Autoscaler job below with your dd_api_key and dd_app_key (if needed, update the DD site as well)
    -  Deploy the Autoscaler job

    --- autoscaler.nomad ---

    job "autoscaler" {
    datacenters = ["dc1"]

    group "autoscaler" {
    count = 1

    network {
    port "http" {}
    }

    task "autoscaler" {
    driver = "docker"

    config {
    image = "hashicorp/nomad-autoscaler-enterprise:0.3.6"
    command = "nomad-autoscaler"
    ports = ["http"]

    args = [
    "agent",
    "-config",
    "local/autoscaler.hcl",
    "-http-bind-address",
    "0.0.0.0",
    "-http-bind-port",
    "${NOMAD_PORT_http}",
    ]
    }

    template {
    destination = "local/autoscaler.hcl"

    data = <<EOH

    nomad {

    // Use Consul service discovery for the Nomad client IP and Port.
    address = "{{ with service "nomad-client" }}{{ with index . 0 }}http://{{.Address}}:{{.Port}}{{ end }}{{ end }}"

    // Use the slapt operator so the autoscaler monitors scaling policies from
    // all Nomad namespaces. If you wish to have it only monitor a single
    // namespace, update this param to match the desired name.
    namespace = "*"

    // If Nomad ALCs are in use, the following line should be uncommented and
    // updated to include an ACL token.
    // token = ""
    }

    // Setup the Prometheus APM so that the autoscaler can pull historical and
    // point-in-time metrics regarding task resource usage.
    apm "prometheus" {
    driver = "prometheus"
    config = {

    // Use Consul service discovery for the Prometheus IP and Port.
    address = "{{ with service "prometheus" }}{{ with index . 0 }}http://{{.Address}}:{{.Port}}{{ end }}{{ end }}"

    // If you wish to use this example without Consul, you can remove the above
    // Consul service discovery address section and replace it with the line
    // below uncommented with an updated endpoint if required.
    // address = "http://localhost:9090"
    }
    }

    // This block contains configuration options specific to the Dynamic Application
    // Sizing enterprise feature.
    dynamic_application_sizing {

    // Lower the evaluate interval so we can reproduce recommendations after only
    // 5 minutes, rather than having to wait for 24hrs as is the default.
    evaluate_after = "5m"
    }

    apm "datadog" {
    config = {
    dd_api_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    dd_app_key = "YYYYYYYYYYYYYYYYYYYYYYYYYYYYY"
    site = "datadoghq.com"
    }
    driver = "datadog"
    }

    policy_eval {

    // Disable the horizontal cluster workers. This
    // helps reduce log noise during the demo.
    workers = {
    cluster = 0
    // horizontal = 0
    }
    }

    policy {
    default_evaluation_interval = "10s"
    default_cooldown = "1m"
    }

    strategy "fixed-value" {
    driver = "fixed-value"
    }

    strategy "pass-through" {
    driver = "pass-through"
    }

    strategy "target-value" {
    driver = "target-value"
    }

    strategy "threshold" {
    driver = "threshold"
    }

    strategy "app-sizing-avg" {
    driver = "app-sizing-avg"
    }

    strategy "app-sizing-percentile" {
    driver = "app-sizing-percentile"
    }

    EOH
    }

    resources {
    cpu = 512
    memory = 512
    }

    service {
    name = "autoscaler"
    port = "http"

    check {
    type = "http"
    path = "/v1/health"
    interval = "5s"
    timeout = "2s"
    }
    }
    }
    }
    }


  • Deploy load test apps for validation testing
    -  There are two jobs used for testing. 
       -  The first job example.nomad contains two task groups:
         1. cache-lb:  This task group is scaled by Dynamic Application Sizing.
         2. cache:  This task group is scaled by Horizontal Autoscaling.
       - The second job loadtest.nomad is a parametrized job that can be executed at any time, causing load to be generated against the example.nomad tasks.

    --- example.nomad ---

    job "example_prom_dd" {
    datacenters = ["dc1"]

    group "cache-lb" {
    count = 1

    network {
    port "lb" {}
    }

    task "nginx" {
    driver = "docker"

    config {
    image = "nginx"
    ports = ["lb"]
    volumes = [
    # It's safe to mount this path as a file because it won't re-render.
    "local/nginx.conf:/etc/nginx/nginx.conf",
    # This path hosts files that will re-render with Consul Template.
    "local/nginx:/etc/nginx/conf.d"
    ]
    }

    # This template overwrites the embedded nginx.conf file so it loads
    # conf.d/*.conf files outside of the `http` block.
    template {
    data = <<EOF
    user nginx;
    worker_processes 1;

    error_log /var/log/nginx/error.log warn;
    pid /var/run/nginx.pid;


    events {
    worker_connections 1024;
    }

    include /etc/nginx/conf.d/*.conf;
    EOF
    destination = "local/nginx.conf"
    }

    # This template creates a TCP proxy to Redis.
    template {
    data = <<EOF
    stream {
    server {
    listen {{ env "NOMAD_PORT_lb" }};
    proxy_pass backend;
    }

    upstream backend {
    {{ range service "redis" }}
    server {{ .Address }}:{{ .Port }};
    {{ else }}server 127.0.0.1:65535; # force a 502
    {{ end }}
    }
    }
    EOF
    destination = "local/nginx/nginx.conf"
    change_mode = "signal"
    change_signal = "SIGHUP"
    }

    resources {
    cpu = 100
    memory = 20
    }

    scaling "cpu" {
    policy {
    cooldown = "1m"
    evaluation_interval = "1m"

    check "95pct" {
    source = "prometheus"
    strategy "app-sizing-percentile" {
    percentile = "95"
    }
    }
    }
    }

    scaling "mem" {
    policy {
    cooldown = "1m"
    evaluation_interval = "1m"

    check "max" {
    source = "prometheus"
    strategy "app-sizing-max" {}
    }
    }
    }
    }

    service {
    name = "redis-lb"
    port = "lb"
    address_mode = "host"

    check {
    type = "tcp"
    port = "lb"
    interval = "10s"
    timeout = "2s"
    }
    }
    }

    group "cache" {
    count = 3

    scaling {
    min = 3
    max = 6
    enabled = true

    policy {
    evaluation_interval = "30s"
    cooldown = "1m"

    check "CPU_400" {
    source = "datadog"
    query = "nomad.client.allocs.cpu.total_ticks{task:redis}"

    strategy "target-value" {
    target = 400
    }
    }
    }
    }

    network {
    port "db" {
    to = 6379
    }
    }

    task "redis" {
    driver = "docker"

    config {
    image = "redis:6.0"
    ports = ["db"]
    }

    resources {
    cpu = 500
    memory = 256
    }

    service {
    name = "redis"
    port = "db"
    address_mode = "host"

    check {
    type = "tcp"
    port = "db"
    interval = "10s"
    timeout = "2s"
    }
    }
    }
    }
    }

    --- loadtest.nomad ---

    job "load-test" {
    datacenters = ["dc1"]
    type = "batch"

    parameterized {
    payload = "optional"
    meta_optional = ["requests", "clients"]
    }

    group "redis-benchmark" {
    task "redis-benchmark" {
    driver = "docker"

    config {
    image = "redis:6.0"
    command = "redis-benchmark"

    args = [
    "-h",
    "${HOST}",
    "-p",
    "${PORT}",
    "-n",
    "${REQUESTS}",
    "-c",
    "${CLIENTS}",
    ]
    }

    template {
    destination = "secrets/env.txt"
    env = true

    data = <<EOF
    {{ with service "redis-lb" }}{{ with index . 0 -}}
    HOST={{.Address}}
    PORT={{.Port}}
    {{- end }}{{ end }}
    REQUESTS={{ or (env "NOMAD_META_requests") "100000" }}
    CLIENTS={{ or (env "NOMAD_META_clients") "50" }}
    EOF
    }

    resources {
    cpu = 100
    memory = 128
    }
    }
    }
    }


Nomad Clusters with ACLs Enabled

If you have enabled ACLs in your Nomad cluster, you'll need to add a token to the autoscaler job before deploying the autoscaler.

There are two token types available that can be used for the autoscaler; a management token with all permissions available or a client token which is more constrained when it comes to permissions.

If you plan to use a client token with the autoscaler, you'll need to ensure that it contains the minimum capabilities needed.  The policy file below contains the minimum capabilities needed by the autoscaler.

 

namespace "*" {
policy = "read"
capabilities = ["read-job", "scale-job", "list-scaling-policies", "read-scaling-policy", "read-job-scaling", "submit-job"]
}

operator {
policy = "read"
}

node {
policy = "read"
}

plugin {
policy = "read"
}

 

Additional Information: 

  • Additional Nomad Datadog metrics can be found here.

  • Additional Prometheus Nomad metrics can be found here.
  • Additional documentation for the Nomad Autoscaler can be found here.

Articles in this section

See more

Related articles