Introduction
There may be times when an autoscaler policy does not perform as expected or fails validation. When this happens, tracing the policy back to it's origin is the first step when troubleshooting the policy. This article covers two options for tracing the policy ID back to the job/group where the policy originated.
Expected Outcome
After reviewing this article, you will quickly be able to trace any policy_id noted in the autoscaler logs back to the job/group where the policy originates from.
Prerequisites (if applicable)
- Access the the Nomad Autoscaler logs
- Access to the Nomad cluster via the Nomad CLI or access to the Nomad API
Use Case
Your autoscaler log shows a particular policy_id which is failing validation.
2023-05-17T15:52:04.156Z [ERROR] policy_manager.policy_handler: policy validation failed: policy_id=6829292f-5a65-550f-9156-b5fd2c22a1fd errors=["scaling.policy is nil", "scaling.policy.check is nil"]
Procedure (Nomad CLI)
-
Make a note of the policy_id in the autoscaler log: policy_id=6829292f-5a65-550f-9156-b5fd2c22a1fd
-
Run "nomad scaling policy info <policy_id>"
Example (using the sample error above):
"nomad scaling policy info 6829292f-5a65-550f-9156-b5fd2c22a1fd" -
The originating job/group is noted in the "Target" field of the output.
Procedure (Nomad API)
-
Make a note of the policy_id in the autoscaler log: policy_id=6829292f-5a65-550f-9156-b5fd2c22a1fd
-
Using curl, query the scaling API "curl https://localhost:4646/v1/scaling/policy/<policy_id>"
Example (using the error noted above):
"curl https://<NOMAD_ADDR>:4646/v1/scaling/policy/6829292f-5a65-550f-9156-b5fd2c22a1fd" -
The originating job/group is noted in the "Target" block of the output.
Additional Information
-
See Command: scaling policy info for additional information and available options.
- See Scaling Policies HTTP API for additional information and available options.