Observability for HashiCorp Nomad with New Relic-Prometheus Integration

Overview

This article explains how to monitor HashiCorp Nomad using New Relic by exporting Nomad metrics in Prometheus format and forwarding them to New Relic using the Prometheus Remote Write Integration.

Prerequisites

• Docker installed and running. Refer this.

• Nomad installed and running. Refer this.

• A valid Nomad Enterprise license file (license.hclic) if using Nomad Enterprise.

1: Configure Nomad Server

1. Create or update the Nomad server configuration file (e.g., /etc/nomad.d/nomad.hcl):

server {
  license_path     = "/opt/license.hclic"
  enabled          = true
  bootstrap_expect = 1
}

bind_addr = "0.0.0.0"

data_dir = "/opt/nomad/data"

advertise {
  http = "1<Server-Pvt-IP>:4646"
  rpc  = "<Server-Pvt-IP>:4647"
  serf = "<Server-Pvt-IP>:4648"
}

ui {
  enabled = true
}

telemetry {
  prometheus_metrics           = true
  publish_allocation_metrics   = true
  publish_node_metrics         = true
}

2. Enable and start the Nomad service:

sudo systemctl enable nomad
sudo systemctl start nomad

3. Verify that Prometheus metrics are exposed:

curl http://localhost:4646/v1/metrics

2: Configure Nomad Client

1. Create or update the Nomad client configuration file (e.g., /etc/nomad.d/nomad.hcl):

data_dir  = "/opt/nomad"
bind_addr = "0.0.0.0"

client {
  enabled = true
  servers = ["<Server-Pvt-IP>:4647"]
}

telemetry {
  prometheus_metrics           = true
  publish_allocation_metrics   = true
  publish_node_metrics         = true
}

2. Enable and start the Nomad client:

sudo systemctl enable nomad
sudo systemctl start nomad

3: Install Prometheus on the Nomad Server

1. Download Prometheus:

cd /opt
sudo wget https://github.com/prometheus/prometheus/releases/download/v3.9.1/prometheus-3.9.1.linux-amd64.tar.gz

2. Extract and move Prometheus:

sudo tar xvf prometheus-3.9.1.linux-amd64.tar.gz
sudo mv prometheus-3.9.1.linux-amd64 prometheus

4: Configure New Relic Prometheus Remote Write

1. Log in to the New Relic UI

2. Navigate to Integrations & Agents

3. Search for Prometheus Remote Write Integration

4. Generate a Remote Write endpoint:

   • Provide a name to identify the Prometheus data source

   • Select an existing API key or create a new one

   • Copy the Remote Write URL and API Key

The connection test may fail at this stage if Prometheus is not yet running. This will be resolved after completing the next steps.

5: Create Prometheus Configuration

1. Edit the Prometheus configuration file:

sudo vi /opt/prometheus/prometheus.yml

2. Add the following configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "nomad"
    metrics_path: /v1/metrics
    params:
      format: ['prometheus']
    static_configs:
      - targets:
          - "<NOMAD_SERVER_PRIVATE_IP>:4646"
          - "<NOMAD_CLIENT_PRIVATE_IP>:4646"

remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=nomad
    headers:
      Api-Key: <NEW_RELIC_LICENSE_KEY>

3. Start Prometheus:

cd /opt/prometheus
sudo ./prometheus --config.file=prometheus.yml

6: Verify Prometheus and New Relic

Verify Prometheus UI

1. Open the Prometheus UI in a browser:

http://<SERVER_PUBLIC_IP>:9090

2. Navigate to:

   • Status → Target Health

   • Confirm that Nomad targets are in UP state

Verify New Relic

1. Return to the New Relic UI

2. Re-run the Test Connection for the Prometheus Remote Write integration

3. Once successful, Nomad metrics and telemetry data will begin appearing in New Relic dashboards and metrics explorer

7: References

https://developer.hashicorp.com/nomad/docs/configuration/telemetry
https://developer.hashicorp.com/nomad/tutorials/manage-clusters/prometheus-metrics