monitoring: document ResourceQuota utilization and stuck-run signals - #1506
Closed
mhotan wants to merge 1 commit into
Closed
monitoring: document ResourceQuota utilization and stuck-run signals#1506mhotan wants to merge 1 commit into
mhotan wants to merge 1 commit into
Conversation
Add a "Dashboards and operational alerts" section covering two operational signals the shipped dashboards surface but the docs did not describe: - ResourceQuota utilization (kube_resourcequota used/hard), why project-quota saturation stalls runs, the dashboard row, and the UnionDPResourceQuotaNearSaturation alert. - Leasor stuck-run diagnostics (leases_by_state, schedule_skip_total, dispatch_total, run-slot saturation, enqueue rejects) and the UnionCPLeasorRunsStuck / UnionCPLeasorEnqueueRejects alerts. Documents the metrics added to the chart dashboards and PrometheusRules. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mhotan
requested review from
EngHabu,
cosmicBboy,
kumare3,
ppiegaze and
samhita-alla
as code owners
August 26, 2026 07:24
GHA build & deploy previewBuilt by
Updated automatically on every push. |
Contributor
Author
|
Superseded by #1511, which targets the self-hosted control-plane metrics glossary ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Documents two operational monitoring signals that the {{< key product_name >}} charts surface in their Grafana dashboards and alert rules but that the monitoring guide did not describe: project ResourceQuota saturation and leasor stuck-run diagnostics. Companion to the helm-charts change that adds these panels and alerts (unionai/helm-charts#560).
Changes
New "Dashboards and operational alerts" section in
deployment/selfmanaged/configuration/monitoring.md:project-quotaResourceQuota, how saturation causesexceeded quotapod rejections that stall runs, thekube_resourcequotautilization query, and theUnionDPResourceQuotaNearSaturationalert.leases_by_state(unassigned vs sent) as the primary stuck signal, plus schedule-skip reasons, dispatch throughput, run-slot saturation, enqueue rejects, and theUnionCPLeasorRunsStuck/UnionCPLeasorEnqueueRejectsalerts.Notes that dashboards render by default (
monitoring.dashboards.enabled) and alerts are opt-in (monitoring.alerting.enabled).Verification
monitoring.alerting.enabled).