feat(metrics): count preemption failures by reason

[Thezone-1]

Description

Two things, both in the preempt action's metrics.

total_preemption_attempts is incremented inside the if succeeded branch, so it counts preemptions that happened, not attempts. Anyone building a success rate from the name gets a wrong answer with no error. It is also redundant: podgroups_scheduled_by_action{action="preempt"} is incremented on the next line under the identical condition, and podgroups_acted_on_by_action{action="preempt"} already counts the attempts. This marks it deprecated in its help text and in METRICS.md and points at that pair. The increment is left where it is, so no existing dashboard changes value. Say the word if you would rather drop it outright or move the increment, and I will do that here instead.

The side that was actually missing is failure with a reason. scenario_search_jobs_total{action="preempt", result} already covers everything that reaches the solver, but a preemptor rejected by IsNonPreemptibleJobOverQueueQuotaFn returns before the solver and produced no row anywhere. This adds preemption_failures_total{reason} and gives that early return its own terminal SearchResult, so every attempt now lands in exactly one of scheduled or failed:

podgroups_acted_on_by_action{action="preempt"}
  == podgroups_scheduled_by_action{action="preempt"} + sum(preemption_failures_total)

The new reason is deliberately not not_attempted. That value is in shouldStopActionForSearchResult's stop set, and returning it there would abandon the preempt action for every job still queued behind this one. There is a test pinning that.

I left the pod-group and queue-level metrics from the issue out of this PR. Those need your call on the cardinality question first, and this half stands on its own.

Related Issues

Fixes #2053

Checklist

• Self-reviewed
• Added/updated tests (if needed)
• Updated documentation (if needed)
• Added a changelog fragment

Breaking Changes

None. total_preemption_attempts keeps its current values and semantics, it is only documented as deprecated.

Additional Notes

I could not run make changelog on this checkout, the Makefile stops with "The OS is not supported" on Windows, so the fragment was written byte-for-byte in the format the non-interactive branch of the target emits. Happy to regenerate it.