adding utilization report to show actions/cpu/gpu/mem - #1450
Open
chrismatteson wants to merge 8 commits into
Open
adding utilization report to show actions/cpu/gpu/mem#1450chrismatteson wants to merge 8 commits into
chrismatteson wants to merge 8 commits into
Conversation
Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
- fix mypy/ty: type the project/domain scope tuples as str and guard the optional client when resolving the console URL in main() - fix ruff: sort imports, wrap >120-char CSS/HTML/JS lines, and encode the intentional NBSP indents and multiplication signs as JS \u escapes - add an 'Export full CSV' button that dumps every collected action row (all raw fields plus core-hours computed from the current assumptions) - add an 'Export view CSV' button that dumps the chart pivot and breakdown table exactly as currently filtered/grouped/stacked Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
The report task OOMKilled (137) on an org where one run alone carried 200k fanned-out sleep actions (1.19M rows total held in memory, then json.dumps'd into the HTML payload — far past the container limit). - aggregate fanouts: identical leaf actions under one parent collapse into a counted row (cnt) with summed durations. Only actions that are nobody's parent aggregate — the run's full action set is listed first, so 'has children' is decidable — keeping drill-down intact while bounding memory and report size. The UI and both CSV exports are cnt-aware, and the tree shows a xN badge on aggregated rows. - crash recovery via traces: the scope list, the run list, and each 50-run sweep batch are @flyte.trace checkpoints inside a collect_actions child task (retries=2), so retries replay completed steps. The run list is traced for correctness, not just speed: batch checkpoints slice it by index, so a re-listed set would misalign them. - OOM escalation: the report task wraps collect_actions in try/except OOMError and re-runs it once at cpu=2/8Gi (the basics/oomer.py pattern); same-size retries can't fix a deterministic OOM, so it is not retried at the collect level. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
Measured against the org that took ~27 min: the server returns a 1000-action page at the same ~1s latency as a 100-action page, so the 200k-action run drops from 2000 sequential round trips to 200. On top of that: - list_actions pages 100 -> 1000, list_runs pages 100 -> 500 - per-scope run listings gather SCOPE_CONCURRENCY (8) at a time - sweep batches gather BATCH_CONCURRENCY (4) at a time — traces gather safely (examples/stress/trace_fanout.py), and results merge in batch order so checkpoint replay alignment is unchanged; one slow batch no longer stalls the pipeline Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
Trace replay only survives retries of the same action, so the OOM escalation path (collect_actions.override(resources=...) — a new action) restarted the sweep from scratch. A flyte.Checkpoint constructed against an explicit URI has no such boundary: - usage_report derives checkpoint_uri from its own raw_data_path and passes it to both the normal sweep and the 8Gi OOM re-run - the blob pins the run list (every restart provably sweeps the same runs — stronger than the traced listing, which re-ran on new actions) and stores each completed batch's aggregated rows - collect_actions restores the blob on startup and only sweeps batches that were still in flight; platform retries (retries=2) resume from the same blob, so the @flyte.trace decorators are gone Validated on dogfood: 1,189,163 actions -> 1,401 rows in 3m07s, and a local round-trip test confirms a fresh Checkpoint on the same URI restores run list + completed batches (the escalation scenario). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
- convert each page of a run's actions to rows as it arrives and drop the page's protos, instead of buffering the whole proto list and converting after — a 200k-action run no longer holds protos and rows at once - retries=1: an OOM burned three same-size attempts before escalating to the bigger container; the checkpoint makes restarts cheap, so one retry is enough transient-crash cover while shortening the OOM detour Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
Every save rewrites the whole blob (runs list + all accumulated rows), so per-batch saves cost quadratic total work as batch counts grow — at a 12-month window (hundreds of batches) the serialized save path was throttling the sweep. Save on a 30s budget instead: a crash loses at most 30s of completed batches, which the resume re-sweeps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
The 12-month sweep succeeded but the run failed at the parent boundary: 'Output file .../outputs.pb exceeds max_bytes limit of 10485760'. Inline task outputs are capped at 10MiB and proto-Struct encoding is several times fatter than JSON, so a decent-size org overflows the cap even after fanout aggregation. collect_actions now writes the rows as compact JSON and returns a flyte.io.File (offloaded to blob storage, no cap); usage_report downloads and parses it before building the report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Matteson <chris.matteson@gmail.com>
cosmicBboy
approved these changes
Sep 2, 2026
cosmicBboy
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.