feat(evals3): benchmark Runs tab as compact table + pass-rate-over-time chart with click-to-filter pills - #479
Conversation
PR Reviewer Guide 🔍(Review updated until commit c4e28e0)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to c4e28e0 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 6cc55d7
Suggestions up to commit 35f21e9
Suggestions up to commit a134cc9
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #479 +/- ##
==========================================
+ Coverage 70.09% 70.19% +0.09%
==========================================
Files 413 416 +3
Lines 34033 34256 +223
Branches 10128 10212 +84
==========================================
+ Hits 23857 24047 +190
- Misses 7915 7937 +22
- Partials 2261 2272 +11
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit 35f21e9 |
VerificationBuilt from this branch's head and run against a real OpenSearch storage cluster ( Unit ( Integration ( E2E ( Scripted interaction check against a real 28-run benchmark (Playwright script, not a test — 3 agents, mixed running/cancelled/completed): Rows measure ~30px tall (were ~140px cards); the blank band between the tab strip and the list is gone (chart top sits <60px below the tabs). CI note: the |
|
Persistent review updated to latest commit 6cc55d7 |
…me chart with click-to-filter pills
Runs tab renders a dense table (Run link · Agent · Model · Size · Pass % ·
Judge · J. Model · Date) under a per-agent pass-rate line chart. Clicking a
categorical cell or a chart legend entry toggles a filter; active filters
show as removable pills (OR within a field, AND across fields). Agent filters
dim the other chart lines instead of dropping them so the legend stays usable
as a toggle. Heat strips move behind a per-row expand chevron. Page padding,
header and row height tightened (~30px rows vs ~140px cards).
Also fixes the ~400px blank band above the runs list: the inactive Cases
TabsContent's 'flex' class out-specified the UA [hidden]{display:none} rule
and kept taking flex-1 space; data-[state=inactive]:hidden restores it.
Repairs tests/unit/components/evals3/BenchmarkRunsPage.test.ts, which had
been failing on 'useLocation is not a function' since the tabs refactor.
Tests: unit (lib + table + page), integration (benchmark GET field contract
on a real backend incl. pagination/polling), e2e (new spec + two re-targeted).
Signed-off-by: goyamegh <goyamegh@amazon.com>
(cherry picked from commit b55a6fc)
…multi-series tooltip (codex review) - buildPassRateSeries orders series alphabetically instead of by point count: series index drives line colour, so busiest-first recoloured an agent mid-poll whenever another agent overtook it. - Latest badge: latestRunId() = max createdAt over the version-filtered merged runs. filteredRuns[0] (pre-existing) missed standalone eval-run docs, which are appended after embedded runs. - Chart tooltip renders every payload entry at the hovered x, not [0]. - Drop the untyped activeDot.onClick → payload.runId navigation and the dead allowDuplicatedCategory prop on a numeric axis. - Pass % cell tooltip states its denominator (judged = passed+failed). Signed-off-by: goyamegh <goyamegh@amazon.com> (cherry picked from commit b41c6de)
…t lib/utils helpers getJudgeModelLabel/getEvaluatorLabel exist only on main-goyamegh (from an open PR), not on origin/main. Inline the two one-liners so the branch applies cleanly to origin/main for the PR. Signed-off-by: goyamegh <goyamegh@amazon.com> (cherry picked from commit 8938ec9)
Signed-off-by: goyamegh <goyamegh@amazon.com>
…r contention The page's loader navigates back to the benchmarks list when GET /api/storage/benchmarks/:id fails (pre-existing behaviour); in CI the seeded benchmark's first GET occasionally timed out under parallel workers, which made the first attempt of one test flake (passed on retry). Route all five tests through openRunsTab(), which retries the navigation once after a bounce and then asserts on the table as before. Signed-off-by: goyamegh <goyamegh@amazon.com> (cherry picked from commit f53ea0a)
6cc55d7 to
c4e28e0
Compare
|
Persistent review updated to latest commit c4e28e0 |
|
Heads-up for reviewers / compose seam with #468 ( This PR and #468 both touch
Resolution used on the integration branch (whichever PR lands second should apply the same):
Verified on the composed tree: |
…h-project#468 (kebab header) and opensearch-project#479 (runs table) This PR was cut from origin/main and merges cleanly against it, but it collides with two sibling PRs in the integration compose: - components/evals3/EvalRunDetailPage.tsx, RunInspectorPage.tsx: opensearch-project#468 trims the lucide import lists (RotateCcw/RotateCw/GitCompare go away with the kebab-only header) and adds `getRunActionVisibility` right after the `computeRunStats` import — the same lines this PR extended with `Ban` and `passRateOverJudged`. Pre-align with opensearch-project#468: leave those lines untouched and add the new symbols on their own import lines anchored on imports neither PR modifies. Zero behaviour change. - components/evals3/BenchmarkRunsPage.tsx: opensearch-project#479 replaces the card list (including `getRunStats` and the per-card stats strip) with the compact `BenchmarkRunsTable`, so this PR's "n not run" cell in that card strip has no home once opensearch-project#479 lands. Drop the edit here (revert the file to origin/main); the table's row builder (`lib/benchmarkRunsTable.ts`) should subtract `notRun` from `pending` as a follow-up on whichever PR lands second. CHANGELOG wording adjusted accordingly. Not pre-aligned (inherently semantic, left for the integration merge): server/routes/storage/evaluationRuns.ts cancel route — opensearch-project#468 rewrote the same handler for the zombie-run fallback; the composed shape is "token present → cancelRequestedAt (this PR) / no token → opensearch-project#468's direct terminal write". Signed-off-by: goyamegh <goyamegh@amazon.com>
Summary
The per-benchmark Runs tab (
/evaluations/benchmarks/:id/runs) was a stack of tall cards — one ~140px card per run with the case-verdict heat strip always expanded — which fit ~3 runs per screen and gave no way to see how agents compare over time. It also had a ~400px blank band above the list (see "Also fixed" below).It is now a compact table with a pass-rate-over-time chart on top:
Running/Cancelledstatus badge, or a chart legend entry toggles a filter. Active filters render as removable pills (Agent: …,Judge: …,J. Model: …) withClearand anN of M runscount. Values on the same field OR together; different fields AND together. Agent filters dim the other chart lines (so a second agent can still be toggled back in from the legend); every other filter narrows the chart to exactly the rows the table shows. Filters are session-scoped by design.p-3/p-4(wasp-4/p-6), headertext-xl, description clamped to 2 lines, same-year dates drop the year. Pass % is colour-graded (≥80 / ≥50 / <50) with thepassed/failed[/⚠errored][/pending]breakdown beside it and a tooltip stating the denominator (judged = passed+failed; errored/pending excluded per Judge/evaluator validation errors are silently reported as completed runs with metrics=0 and a misleading reason #242).Existing behaviour preserved: Compare checkbox selection (Select All now acts on the visible/filtered rows), Delete/Cancel gating for standalone eval-run docs, version filter + its empty state,
Latestbadge, polling while a run is in flight.Also fixed
TabsContentwith thehiddenattribute, but theflex flex-colclass on the Cases panel (needed for Benchmark case review: Cases/Runs tabs, master–detail with suite health, per-run heat strips #447's scrolling fix) out-specifies the UA[hidden]{display:none}rule — so the inactive Cases panel stayeddisplay:flexwithflex-1and pushed the Runs panel ~400px down.data-[state=inactive]:hiddenfixes it.Latestbadge wasfilteredRuns[0]; the merged list appends standalone eval-run docs after embedded runs, so a newer CLI/API-started run never got it. NowlatestRunId()= maxcreatedAt.tests/unit/components/evals3/BenchmarkRunsPage.test.tsgets the fuller router mock it needs (useLocation/Link) and drops the coarseCaseHeatStripstub (the heat-strip expand test needs the real one; the ESMreact-markdownchain is neutralised via the@/components/ui/markdownmock instead).Tests
tests/unit/lib/benchmarkRunsTable.test.ts(22)latestRunIdtests/unit/components/evals3/BenchmarkRunsTable.test.ts(11)tests/unit/components/evals3/BenchmarkRunsPage.test.ts(15)Latestvia createdAt, filter pills, legend toggle, heat-strip expand, inactive-panel regressiontests/integration/server/routes/storage/benchmarkRunsTableFields.integration.test.ts(4, real backend)GET /api/storage/benchmarks/:idround-tripsjudgeModelId/evaluatorId/testCaseSnapshots/per-result verdicts the columns depend on, incl. under?runsSize=pagination andfields=polling; stalerun.statsignoredtests/e2e/benchmark-runs-table.spec.ts(5, new)benchmarkruns-associated-evalruns.spec.ts,benchmark-runs-passed-count.spec.tsdata-testidsAll of the above green locally against a server built from this branch's head, wired to a real OpenSearch cluster (test data tracked + cleaned by id).
Adversarial review (
codex_review, gpt-5.4)Applied: stable chart colours under polling (series were ordered by point count) ·
LatestbycreatedAt· tooltip renders every series at the hovered x (waspayload[0]) · dropped the untypedactiveDot.onClick → payload.runIdnavigation and a deadallowDuplicatedCategoryprop on a numeric axis · Pass % denominator tooltip.Rejected: including errored cases in the Pass % denominator (contradicts the repo-wide #242 convention every other surface follows) · making agent pills narrow the chart like other fields (deliberate — keeps the legend usable as a toggle, documented inline) · removing row-click navigation (pre-existing, covered by existing e2e) · folding the
runningback-solve intocomputeRunStats(moved verbatim from the previous renderer;computeRunStatsdoesn't exposerunning).Checklist
CHANGELOG.mdupdated under## [Unreleased]npm run build:all·npm audit --audit-level=highclean