[API] Add provisioning/pending/start-time metrics and availability dashboard panels - #10625
Open
lloyd-brown wants to merge 3 commits into
Open
[API] Add provisioning/pending/start-time metrics and availability dashboard panels#10625lloyd-brown wants to merge 3 commits into
lloyd-brown wants to merge 3 commits into
Conversation
…nels
Expose three new lifecycle metrics from the API server:
- sky_provision_duration_seconds{cloud,result}: wall-clock time of each
provisioning attempt (compute acquisition time), observed in
bulk_provision around the existing timing that was previously only
logged. Failure attempts are observed before teardown starts.
- sky_apiserver_request_pending_seconds{name,schedule_type}: time from
request creation to its first execution start. Unlike
sky_apiserver_queue_wait_seconds (per-enqueue queue residency), this
includes scheduling preconditions and retry backoff, and is observed
exactly once per request (guarded on pid being unset) so retry-requeue
loops cannot re-observe an ever-growing age (see #9988).
- sky_apiserver_start_time_seconds: server boot timestamp, sourced from
the main server process creation time. The boot-check request row is
deliberately not used: it survives fast server restarts (the scheduler
swallows RequestAlreadyExistsError), so its created_at can reflect an
older boot.
Dashboard: add a Scheduling & Provisioning row (request pending P95,
provision duration P95/avg) and a Server Availability stat to the API
server overview dashboard, and rework the top-row layout to fit it. The
availability stat shows the dashboard-range value and a fixed
trailing-30d value backed by new documented recording rules; docs cover
the rule group, retention requirements, and the caveats of quoting the
30d number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
- first-execution detection for sky_apiserver_request_pending_seconds now keys on status == PENDING instead of pid: the ExecutionRetryableError handler clears pid before requeueing, so every retry looked like a first run. The retry/pause path is the only writer of WAITING, so PENDING at execution start identifies the first attempt exactly. - ExecutionPausedError no longer records a provision failure: a pause to wait on an external condition is neither success nor failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DanielZhangQD
left a comment
DanielZhangQD
left a comment
Collaborator
There was a problem hiding this comment.
Thanks! @lloyd-brown
… test
- Availability panel selects up{app=~"$app", app!=""} alongside the job
regex, so annotation-discovered deployments match regardless of release
name; app!="" keeps an unresolved $app from matching every target. The
documented recording rule uses an explicit <release>-api placeholder.
- Docs no longer suggest server.retention (the chart already defaults to
1000d; the 15d default is upstream Prometheus's, noted as such).
- Pending-metric wording corrected everywhere: observed at the first
execution start, so retry backoff after that start is excluded;
preconditions are included. Guard extracted into
_maybe_observe_request_pending with a unit test pinning both
directions (PENDING observes once, WAITING re-execution observes
nothing).
- Cancellation (KeyboardInterrupt/SystemExit) no longer recorded as a
provisioning failure.
- Dashboard second band is four 6-wide panels, removing the 16x8 gap
left by the RPS-by-User overlap fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sky_provision_duration_seconds{cloud,result}: wall-clock time of each provisioning attempt (compute acquisition time), observed around_bulk_provisionwhere the duration was previously only logged. Failures are observed before failover teardown starts, so teardown time is not counted.sky_apiserver_request_pending_seconds{name,schedule_type}: time from request creation to its first execution start. Complementssky_apiserver_queue_wait_seconds(per-enqueue queue residency) by including scheduling preconditions and retry backoff. Observed exactly once per request (guarded onpidbeing unset), so a--retry-until-uprequest looping through the retry-requeue path cannot repeatedly re-observe its ever-growing age (the same failure mode [API] Measure executor queue-wait per-enqueue, not since request creation #9988 fixed for the queue-wait metric).sky_apiserver_start_time_seconds: server boot timestamp (uptime =time() - metric). Sourced from the main server process creation time via a custom collector;process_start_time_secondsis unavailable under the multiprocess collector. The boot-check request row (which debug dumps use for uptime) is deliberately not used:schedule_on_boot_check_asyncswallowsRequestAlreadyExistsError, so that row survives fast restarts with a stalecreated_at.api-server-overview.json): new Scheduling & Provisioning row (Request Pending P95 by request name; Provision Duration P95 + average by cloud/result) and a Server Availability stat in the reworked top row showing both the dashboard-range availability and a fixed trailing-30-day value. Also fixes a grid overlap between the RPS by User and Request duration panels.api-server-metrics-setup.rst): documents the new metrics, and adds an Availability tracking section with theskypilot-slorecording-rule group backing the 30d value (via the Prometheus subchart'sserverFiles), the retention requirement (>= 30d; Prometheus defaults to 15d), and the caveats of quoting the 30d number (no backfill; unmeasured time is not counted;upmeasures metrics-endpoint reachability, not request success).Test plan
Ran a local API server (Docker, Postgres,
SKY_API_SERVER_METRICS_ENABLED=true, multiprocess mode) with Prometheus + Grafana scraping/metrics:sky_provision_duration_seconds_count{cloud="Kubernetes",result="success"}incremented with plausible sums (~18s first launch, ~5s warm relaunch); P95/avg panel queries return data.sky status/sky jobs queueand internal daemons:sky_apiserver_request_pending_secondspopulated for both the subprocess executor path and the coroutine path, labeled by request name; verifiedhistogram_quantile(0.95, ...)in Prometheus.sky_apiserver_start_time_secondsmatches the new boot within seconds (the boot-check-row approach was verified stale across restarts, which motivated the process-creation-time source).sky:apiserver_up:maxcounted the expected number of down-samples.bash format.sh --files <changed files>: yapf/isort clean, mypy clean, pylint 10.00/10.🤖 Generated with Claude Code