Introduction
In a dynamic orchestration environment like HashiCorp Nomad, tracking the lifecycle of allocations—including their creation time, start time, and runtime duration—is critical for operational visibility, troubleshooting, and performance optimization. However, Nomad does not natively expose these timestamps as built-in Prometheus metrics, making it challenging to monitor allocation lifespan directly through standard observability tools.
Expected Outcome
By following the methods outlined in this Knowledge Base, you will achieve the following:
Visibility into Allocation Lifecycle Events
Track allocation creation time – Know exactly when an allocation was scheduled and deployed.
Monitor task start time – Detect delays in container initialization, image pulls, or dependency readiness.
Measure runtime duration – Identify long-running or stale allocations that may need recycling.
Prerequisites (if applicable)
Nomad Cluster – Running and accessible.
Nomad CLI – Installed and configured.
API Access – Permission to query /v1/allocations.jq & curl – For parsing JSON and API calls.
Cause
Nomad's metrics focus on real-time resource usage rather than allocation lifecycle events. While creation/start times are available in the API, they aren't exposed as built-in metrics, requiring custom solutions for tracking allocation age and scheduling performance.
Use Case
Below are potential solutions to address these issues on macOS.
- I have run a sample example job with 3 allocations.
vaibhavarora@vaibhavarora-FPYJ49Y7NR kind % nomad job status example ID = example Name = example Submit Date = 2025-06-17T09:46:39+05:30 Type = service Priority = 50 Datacenters = * Namespace = default Node Pool = default Status = running Periodic = false Parameterized = false Summary Task Group Queued Starting Running Failed Complete Lost Unknown cache 0 0 3 0 0 0 0 Latest Deployment ID = bd33e1d7 Status = successful Description = Deployment completed successfully Deployed Task Group Desired Placed Healthy Unhealthy Progress Deadline cache 3 3 3 0 2025-06-17T09:56:50+05:30 Allocations ID Node ID Task Group Version Desired Status Created Modified 7ee714b0 10c54b6d cache 0 run running 4h1m ago 4h1m ago cfbdc686 10c54b6d cache 0 run running 4h1m ago 4h1m ago d06b28ed 10c54b6d cache 0 run running 4h1m ago 4h1m ago vaibhavarora@vaibhavarora-FPYJ49Y7NR kind %
2. To check the allocation creation time, which also indicates how long the allocation has been running.
The Allocation Creation Time is the timestamp when the Nomad scheduler successfully assigns a workload (allocation) to a client node.
curl -s http://localhost:4646/v1/allocations | jq '.[] | {ID, CreateTime}'
{
"ID": "7ee714b0-8214-649e-e4fb-5ddcefea2d67",
"CreateTime": 1750133799053997000
}
{
"ID": "cfbdc686-a338-e56d-ab6c-22fbb1c44505",
"CreateTime": 1750133799053997000
}
{
"ID": "d06b28ed-1683-99bc-c4f1-676fce4696d2",
"CreateTime": 1750133799053997000
}
Command to convert Creation Time to a readable timestamp:
curl -s http://localhost:4646/v1/allocations | jq '.[] | {ID, CreateTime: (.CreateTime / 1000000000 | todate)}'
{
"ID": "7ee714b0-8214-649e-e4fb-5ddcefea2d67",
"CreateTime": "2025-06-17T04:16:39Z"
}
{
"ID": "cfbdc686-a338-e56d-ab6c-22fbb1c44505",
"CreateTime": "2025-06-17T04:16:39Z"
}
3. To check the allocation start time,Task "redis" is "running". We need to pass the allocation ID after v1/allocations.
The Allocation Start Time refers to the timestamp when a task within a Nomad allocation actually begins execution on the client node.
vaibhavarora@vaibhavarora-FPYJ49Y7NR kind % curl -s http://localhost:4646/v1/allocation/7ee714b0-8214-649e-e4fb-5ddcefea2d67 | jq '.TaskStates.redis.StartedAt' "2025-06-17T04:16:40.29791Z"
4. Get all allocation IDs:
nomad alloc status -json | jq -r '.[].ID' | while read alloc_id; do
curl -s "http://localhost:4646/v1/allocation/$alloc_id" | \
jq --arg id "$alloc_id" '{ID: $id, StartedAt: .TaskStates.redis.StartedAt}'
done
{
"ID": "7ee714b0-8214-649e-e4fb-5ddcefea2d67",
"StartedAt": "2025-06-17T04:16:40.29791Z"
}
{
"ID": "cfbdc686-a338-e56d-ab6c-22fbb1c44505",
"StartedAt": "2025-06-17T04:16:40.314281Z"
}
{
"ID": "d06b28ed-1683-99bc-c4f1-676fce4696d2",
"StartedAt": "2025-06-17T04:16:40.283718Z"
}
Outcome
This KB enables precise tracking of Nomad allocation lifecycles, including IST timestamps and runtime metrics. It provides CLI and API solutions to monitor scheduling delays, detect slow starts, and integrate with observability tools - transforming raw data into actionable performance insights for better cluster management.
I've submitted a feature request to the Nomad engineering team to include allocation lifecycle metrics (creation time and start time) as native Prometheus metrics. This enhancement is planned for a future Nomad release.
Additional Information