Introduction:

HashiCorp Nomad is a powerful workload orchestrator that allows you to deploy and manage applications across a cluster of servers. Integrating Nomad with Prometheus, a popular monitoring and alerting toolkit, can provide deep insights into your Nomad cluster's performance and health. This article provides a detailed step-by-step guide to integrating Nomad with Prometheus.

Prerequisites:

1. Nomad Cluster: A running Nomad cluster with jobs you want to monitor.
2. Prometheus: Prometheus installed and configured. You can download Prometheus from the official website: Prometheus Downloads.
3. Nomad ACL Token: If ACLs are enabled in your Nomad cluster, you'll need an ACL token with sufficient permissions for Prometheus to access Nomad's API.

Step 1: Configure Consul agents with Prometheus:

1. Navigate to the consul agent configuration path and enable "telemetry" by using the below parameters - 

telemetry {
  collection_interval = "1s"
  disable_hostname = true
  prometheus_metrics = true
  publish_allocation_metrics = true
  publish_node_metrics = true
}

You can use other parameters as well related to telemetry. Here is the URL for the reference - 

https://developer.hashicorp.com/nomad/docs/configuration/telemetry

2. Once the telemetry parameters are set, reload the nomad configuration.

Step 2: Configure Prometheus on Nomad nodes -

1. Once you installed the Prometheus on your Nomad nodes, open the Prometheus configuration file (prometheus.yml) and add a new scrape configuration for Consul metrics monitoring -

  - job_name: 'nomad'
    metrics_path: "/v1/metrics"
    bearer_token: "******-******-******-******"
    params:
      format: ['prometheus']
    scheme: http
    static_configs:
    - targets: ['127.0.0.1:4646']

In the above, please adjust as per your environment -

• Replace bearer_token with your Nomad ACL token with enough permissions.
• Replace metrics_path with the name of the service you want to monitor.
• Replace targets with your Nomad HTTP address.

2. Save the file and restart your Prometheus services.

Step 3: Verify Prometheus Configuration

1. Access the Prometheus web interface at http://localhost:9090.
2. Navigate to the "Targets" page to ensure that Prometheus successfully scrapes metrics from Nomad Cluster.
3. Below are some sample snippets for your reference -

Snippet 1 - Targets section under Prometheus dashboard -

Snippet 2 - Graph section for Nomad's nomad.memberlist.gossip section.

Snippet 3 - Graph section for Nomad's nomad.client.allocations.running section -

Conclusion -

By following these steps, you can integrate HashiCorp Nomad with Prometheus to effectively monitor your Nomad cluster. This integration provides you with valuable insights into your cluster's performance and health, helping you ensure optimal application deployment and management.

Reference -

NA