Skip to content

feat(evals3): benchmark Runs tab as compact table + pass-rate-over-time chart with click-to-filter pills - #479

Open
goyamegh wants to merge 5 commits into
opensearch-project:mainfrom
goyamegh:goyamegh/runs-table-graph-pr
Open

feat(evals3): benchmark Runs tab as compact table + pass-rate-over-time chart with click-to-filter pills#479
goyamegh wants to merge 5 commits into
opensearch-project:mainfrom
goyamegh:goyamegh/runs-table-graph-pr

Conversation

@goyamegh

@goyamegh goyamegh commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

The per-benchmark Runs tab (/evaluations/benchmarks/:id/runs) was a stack of tall cards — one ~140px card per run with the case-verdict heat strip always expanded — which fit ~3 runs per screen and gave no way to see how agents compare over time. It also had a ~400px blank band above the list (see "Also fixed" below).

It is now a compact table with a pass-rate-over-time chart on top:

  • Table columns: Run (link to the inspector) · Agent · Model · Size · Pass % · Judge (evaluator) · J. Model (judge model) · Date. Rows are ~30px. Every column sorts (newest-first default; null pass rates always sink).
  • Chart: one line per agent, real time X-axis, 0–100% Y-axis. Legend entries are clickable.
  • Click-to-filter with pills: clicking any Agent / Model / Judge / J. Model cell, a Running/Cancelled status badge, or a chart legend entry toggles a filter. Active filters render as removable pills (Agent: …, Judge: …, J. Model: …) with Clear and an N of M runs count. Values on the same field OR together; different fields AND together. Agent filters dim the other chart lines (so a second agent can still be toggled back in from the legend); every other filter narrows the chart to exactly the rows the table shows. Filters are session-scoped by design.
  • Compact: page padding p-3/p-4 (was p-4/p-6), header text-xl, description clamped to 2 lines, same-year dates drop the year. Pass % is colour-graded (≥80 / ≥50 / <50) with the passed/failed[/⚠errored][/pending] breakdown beside it and a tooltip stating the denominator (judged = passed+failed; errored/pending excluded per Judge/evaluator validation errors are silently reported as completed runs with metrics=0 and a misleading reason #242).
  • Heat strips are still there — behind a per-row expand chevron instead of always occupying space.

Existing behaviour preserved: Compare checkbox selection (Select All now acts on the visible/filtered rows), Delete/Cancel gating for standalone eval-run docs, version filter + its empty state, Latest badge, polling while a run is in flight.

Also fixed

  • Blank band above the runs list. Radix hides the inactive TabsContent with the hidden attribute, but the flex flex-col class on the Cases panel (needed for Benchmark case review: Cases/Runs tabs, master–detail with suite health, per-run heat strips #447's scrolling fix) out-specifies the UA [hidden]{display:none} rule — so the inactive Cases panel stayed display:flex with flex-1 and pushed the Runs panel ~400px down. data-[state=inactive]:hidden fixes it.
  • Latest badge was filteredRuns[0]; the merged list appends standalone eval-run docs after embedded runs, so a newer CLI/API-started run never got it. Now latestRunId() = max createdAt.
  • tests/unit/components/evals3/BenchmarkRunsPage.test.ts gets the fuller router mock it needs (useLocation/Link) and drops the coarse CaseHeatStrip stub (the heat-strip expand test needs the real one; the ESM react-markdown chain is neutralised via the @/components/ui/markdown mock instead).

Tests

Level File Covers
Unit tests/unit/lib/benchmarkRunsTable.test.ts (22) pass-rate derivation excludes errored/pending; filter AND/OR + toggle; sort incl. null-sinking; per-agent series ordering is stable under new runs; un-judged runs dropped from chart; latestRunId
Unit tests/unit/components/evals3/BenchmarkRunsTable.test.ts (11) cell click filters without navigating; pill labels; action gating; expand; aria-sort
Unit tests/unit/components/evals3/BenchmarkRunsPage.test.ts (15) columns, default sort, Latest via createdAt, filter pills, legend toggle, heat-strip expand, inactive-panel regression
Integration tests/integration/server/routes/storage/benchmarkRunsTableFields.integration.test.ts (4, real backend) GET /api/storage/benchmarks/:id round-trips judgeModelId/evaluatorId/testCaseSnapshots/per-result verdicts the columns depend on, incl. under ?runsSize= pagination and fields=polling; stale run.stats ignored
E2E tests/e2e/benchmark-runs-table.spec.ts (5, new) headers exactly as designed; chart above table with one legend entry per agent; rows ≤40px and no blank band; click→pill→remove; AND/OR + legend toggling + Clear; run link → inspector; heat-strip expand
E2E benchmarkruns-associated-evalruns.spec.ts, benchmark-runs-passed-count.spec.ts re-targeted from card selectors to the table's data-testids

All of the above green locally against a server built from this branch's head, wired to a real OpenSearch cluster (test data tracked + cleaned by id).

Adversarial review (codex_review, gpt-5.4)

Applied: stable chart colours under polling (series were ordered by point count) · Latest by createdAt · tooltip renders every series at the hovered x (was payload[0]) · dropped the untyped activeDot.onClick → payload.runId navigation and a dead allowDuplicatedCategory prop on a numeric axis · Pass % denominator tooltip.
Rejected: including errored cases in the Pass % denominator (contradicts the repo-wide #242 convention every other surface follows) · making agent pills narrow the chart like other fields (deliberate — keeps the legend usable as a toggle, documented inline) · removing row-click navigation (pre-existing, covered by existing e2e) · folding the running back-solve into computeRunStats (moved verbatim from the previous renderer; computeRunStats doesn't expose running).

Checklist

  • DCO signoff on every commit
  • CHANGELOG.md updated under ## [Unreleased]
  • npm run build:all · npm audit --audit-level=high clean
  • Unit + integration + e2e added (see table)
  • SPDX headers on new files

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit c4e28e0)

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ Recommended focus areas for review

Possible Issue

When minT === maxT (all runs at the same instant), the code sets pad = 86_400_000 (one day) but then computes ticks with computeTimeTicks(minT, maxT) using the original un-padded bounds. If the single point falls on a day boundary, computeTimeTicks may produce ticks that lie outside [minT - pad, maxT + pad], causing recharts to render them off-canvas or not at all. The domain padding is applied to the XAxis but the tick computation ignores it.

const allT = series.flatMap(s => s.points.map(p => p.t));
const minT = Math.min(...allT);
const maxT = Math.max(...allT);
// A single point (or all runs in one instant) needs a non-zero domain or
// recharts collapses the axis; pad by a day on each side.
const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 86_400_000;
const { ticks, dense } = computeTimeTicks(minT, maxT);
const tickTime = makeTickFormatter(dense);
Possible Issue

computePassRate rounds (passed / evaluable) * 1000 / 10 to produce a percent with one decimal. When passed = 1 and evaluable = 3, this yields 33.3. However, 1/3 * 1000 = 333.333..., and Math.round(333.333...) = 333, so the result is 33.3. But 2/3 * 1000 = 666.666... rounds to 667, giving 66.7. The asymmetry means 1/3 + 2/3 = 100% in the UI but the individual rates are 33.3% + 66.7% = 100% only by coincidence. A user filtering to subsets may see rates that don't sum as expected. Consider rounding the final percentage (e.g., Math.round((passed / evaluable) * 100 * 10) / 10) instead of the intermediate product.

export function computePassRate(
  passed: number, failed: number,
): number | null {
  const evaluable = passed + failed;
  if (evaluable <= 0) return null;
  return Math.round((passed / evaluable) * 1000) / 10;
}
Possible Issue

handleToggleSelectAll checks visibleRows.every(r => selectedRunIds.includes(r.run.id)) to decide whether to select or deselect all. If the user has selected runs that are now hidden by a filter, those selections persist in selectedRunIds but are not in visibleRows. Clicking "Select All" when all visible rows are already selected will deselect them, but the hidden selections remain. Clicking "Deselect All" afterward has no visible effect because visibleRows is empty of selections, leaving the hidden selections orphaned. The Compare button's disabled={selectedRunIds.length < 2} can stay enabled with only hidden runs selected, and clicking it will attempt to compare runs the user cannot see in the table.

const handleToggleSelectAll = () => {
  const allRunIds = visibleRows.map(r => r.run.id);
  const allSelected = allRunIds.every(id => selectedRunIds.includes(id));
  setSelectedRunIds(allSelected ? [] : allRunIds);
};

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

PR Code Suggestions ✨

Latest suggestions up to c4e28e0

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Prevent selection clearing on empty rows

When visibleRows is empty (e.g., all runs filtered out), allRunIds.every(...)
returns true for an empty array, causing the "Select All" button to show "Deselect
All" and clearing any previously selected runs from other filter states. Guard
against empty visible rows to preserve selection state.

components/evals3/BenchmarkRunsPage.tsx [536-540]

 const handleToggleSelectAll = () => {
   const allRunIds = visibleRows.map(r => r.run.id);
+  if (allRunIds.length === 0) return;
   const allSelected = allRunIds.every(id => selectedRunIds.includes(id));
   setSelectedRunIds(allSelected ? [] : allRunIds);
 };
Suggestion importance[1-10]: 7

__

Why: Valid bug fix. When visibleRows is empty due to filters, allRunIds.every(...) returns true (vacuous truth), causing the button to incorrectly show "Deselect All" and clear selections. The guard prevents this edge case and preserves user intent.

Medium
Ensure minimum axis padding threshold

When all runs occur at the same instant (minT === maxT), the padding is set to
exactly one day (86_400_000 ms). However, if runs span less than one day but are not
identical, the 4% padding may be insufficient for recharts to render a readable
axis. Consider using a minimum padding threshold (e.g., 1 hour) to ensure the axis
remains legible for tightly clustered runs.

components/evals3/BenchmarkPassRateChart.tsx [107]

-const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 86_400_000;
+const MIN_PAD = 3_600_000; // 1 hour
+const pad = maxT - minT > 0 ? Math.max((maxT - minT) * 0.04, MIN_PAD) : 86_400_000;
Suggestion importance[1-10]: 5

__

Why: The suggestion correctly identifies that tightly clustered runs (spanning less than a day but not identical) might have insufficient padding with 4%. Adding a minimum threshold improves axis readability, though the impact is moderate since the existing fallback (one day for identical timestamps) already handles the most extreme case.

Low
Validate non-negative input parameters

The function does not validate that passed and failed are non-negative numbers. If
negative values are passed (e.g., from corrupted data or computation errors), the
pass rate calculation will produce incorrect results. Add input validation to ensure
both parameters are non-negative before computing the rate.

lib/benchmarkRunsTable.ts [63-69]

 export function computePassRate(
   passed: number, failed: number,
 ): number | null {
+  if (passed < 0 || failed < 0) return null;
   const evaluable = passed + failed;
   if (evaluable <= 0) return null;
   return Math.round((passed / evaluable) * 1000) / 10;
 }
Suggestion importance[1-10]: 4

__

Why: While defensive validation is good practice, the suggestion addresses a hypothetical scenario (corrupted data producing negative counts) that is unlikely given the upstream computeRunStats logic. The improvement is marginal and the function already handles the zero/negative evaluable case correctly.

Low

Previous suggestions

Suggestions up to commit 6cc55d7
CategorySuggestion                                                                                                                                    Impact
General
Reduce single-point chart padding

The fallback padding of exactly one day (86,400,000 ms) when all runs occur at the
same instant may cause the chart to display an unnecessarily wide time range.
Consider using a smaller padding value or making it proportional to a typical run
duration to avoid misleading time scales.

components/evals3/BenchmarkPassRateChart.tsx [107]

-const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 86_400_000;
+const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 3_600_000; // 1 hour padding for single-point case
Suggestion importance[1-10]: 4

__

Why: The suggestion to reduce padding from 1 day to 1 hour for single-point cases is reasonable for improving chart readability, but the current 1-day padding is a deliberate design choice that works. The impact is minor since single-point scenarios are edge cases.

Low
Remove default parameter for consistency

The function defaults now to new Date() on every call, which can cause inconsistent
behavior during testing or when the function is called multiple times near midnight
on New Year's Eve. Consider accepting now as a required parameter or using a stable
reference time.

components/evals3/BenchmarkRunsTable.tsx [129-137]

-export function formatRunDate(iso: string, now: Date = new Date()): string {
+export function formatRunDate(iso: string, now: Date): string {
   const d = new Date(iso);
   if (Number.isNaN(d.getTime())) return '—';
   const sameYear = d.getFullYear() === now.getFullYear();
   return d.toLocaleString('en-US', {
     month: 'short', day: 'numeric', ...(sameYear ? {} : { year: 'numeric' }),
     hour: '2-digit', minute: '2-digit',
   });
 }
Suggestion importance[1-10]: 3

__

Why: The concern about inconsistent behavior near midnight on New Year's Eve is valid but extremely edge-case. The default parameter now = new Date() is a common pattern for testability. The suggestion would require all call sites to pass now, reducing convenience without significant benefit.

Low
Tighten tick boundary conditions

The loop condition t <= maxT + step combined with the filter t >= minT - step can
produce ticks outside the intended range. Verify that the tick generation logic
correctly bounds ticks to the visible domain, especially at the edges.

components/evals3/BenchmarkPassRateChart.tsx [44-46]

-export function computeTimeTicks(minT: number, maxT: number): { ticks: number[]; dense: boolean } {
-  if (!Number.isFinite(minT) || !Number.isFinite(maxT) || maxT < minT) return { ticks: [], dense: false };
-  const span = maxT - minT;
-  const dense = span < 2 * DAY_MS;
-  const step = dense ? 6 * 3_600_000 : DAY_MS;
-  const start = new Date(minT);
-  if (dense) start.setMinutes(0, 0, 0); else start.setHours(0, 0, 0, 0);
-  const ticks: number[] = [];
-  for (let t = start.getTime(); t <= maxT + step; t += step) {
-    if (t >= minT - step) ticks.push(t);
-  }
+for (let t = start.getTime(); t <= maxT; t += step) {
+  if (t >= minT) ticks.push(t);
+}
Suggestion importance[1-10]: 2

__

Why: The suggestion misunderstands the intent of the padding logic (minT - step and maxT + step). The current implementation deliberately extends ticks slightly beyond the data range to ensure proper chart rendering. The proposed change would break this intentional behavior and potentially cause edge ticks to be missing.

Low
Suggestions up to commit 35f21e9
CategorySuggestion                                                                                                                                    Impact
General
Remove default parameter for consistency

The function defaults now to new Date() on every call, which can cause inconsistent
year comparisons if the function is called around midnight on New Year's Eve or if
the system clock changes. Pass now explicitly from the caller to ensure consistent
date formatting across all rows in a single render.

components/evals3/BenchmarkRunsTable.tsx [129-137]

-export function formatRunDate(iso: string, now: Date = new Date()): string {
+export function formatRunDate(iso: string, now: Date): string {
   const d = new Date(iso);
   if (Number.isNaN(d.getTime())) return '—';
   const sameYear = d.getFullYear() === now.getFullYear();
   return d.toLocaleString('en-US', {
     month: 'short', day: 'numeric', ...(sameYear ? {} : { year: 'numeric' }),
     hour: '2-digit', minute: '2-digit',
   });
 }
Suggestion importance[1-10]: 6

__

Why: The concern about inconsistent year comparisons across midnight on New Year's Eve is valid but extremely rare. Removing the default parameter would require all callers to pass now explicitly, which is a reasonable consistency improvement. The existing test already passes now explicitly, suggesting the default may be unnecessary.

Low
Constrain tick generation to data range

The loop condition t <= maxT + step combined with the push condition t >= minT -
step can generate ticks outside the intended [minT, maxT] range, potentially causing
axis labels to appear beyond the data domain. Tighten the loop bounds to ensure
ticks stay within or very close to the actual data range.

components/evals3/BenchmarkPassRateChart.tsx [44-46]

-export function computeTimeTicks(minT: number, maxT: number): { ticks: number[]; dense: boolean } {
-  if (!Number.isFinite(minT) || !Number.isFinite(maxT) || maxT < minT) return { ticks: [], dense: false };
-  const span = maxT - minT;
-  const dense = span < 2 * DAY_MS;
-  const step = dense ? 6 * 3_600_000 : DAY_MS;
-  const start = new Date(minT);
-  if (dense) start.setMinutes(0, 0, 0); else start.setHours(0, 0, 0, 0);
-  const ticks: number[] = [];
-  for (let t = start.getTime(); t <= maxT + step; t += step) {
-    if (t >= minT - step) ticks.push(t);
-  }
-  ...
+for (let t = start.getTime(); t <= maxT; t += step) {
+  if (t >= minT) ticks.push(t);
 }
Suggestion importance[1-10]: 5

__

Why: The current logic intentionally includes ticks slightly outside [minT, maxT] (via minT - step and maxT + step) to ensure edge data points aren't clipped by axis bounds. The suggestion would tighten this but may cause visual issues. The existing approach is deliberate, though the suggestion identifies a potential refinement.

Low
Reduce single-point chart padding

When all runs occur at the same instant (maxT === minT), the fixed 1-day padding may
be excessive for short-lived benchmarks or cause visual confusion. Consider using a
smaller default padding (e.g., 1 hour = 3,600,000 ms) or making it proportional to a
typical run interval to avoid an unnecessarily wide time axis.

components/evals3/BenchmarkPassRateChart.tsx [107]

-const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 86_400_000;
+const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 3_600_000;
Suggestion importance[1-10]: 4

__

Why: The suggestion to reduce padding from 1 day to 1 hour for single-point charts is reasonable for visual clarity, but the impact is minor since this is an edge case (all runs at the same instant). The current 1-day padding is not incorrect, just potentially excessive.

Low
Validate non-negative pass/fail counts

The function does not validate that passed and failed are non-negative integers.
Negative or non-integer inputs could produce incorrect pass rates. Add input
validation to ensure both parameters are non-negative numbers before computing the
rate.

lib/benchmarkRunsTable.ts [63-69]

 export function computePassRate(
   passed: number, failed: number,
 ): number | null {
+  if (passed < 0 || failed < 0) return null;
   const evaluable = passed + failed;
   if (evaluable <= 0) return null;
   return Math.round((passed / evaluable) * 1000) / 10;
 }
Suggestion importance[1-10]: 3

__

Why: Adding validation for non-negative inputs is a defensive programming practice, but passed and failed are derived from computeRunStats which already ensures non-negative counts. The validation would catch programming errors rather than user input issues, making it a low-priority enhancement.

Low
Suggestions up to commit a134cc9
CategorySuggestion                                                                                                                                    Impact
General
Align dense-mode ticks to 6-hour boundaries

The setMinutes(0, 0, 0) call in dense mode only zeroes minutes/seconds/milliseconds
but leaves the hour unchanged. For a 6-hour step, the first tick should align to a
6-hour boundary (0, 6, 12, 18), not just round down to the current hour. This can
produce misleading tick labels when the data starts mid-interval.

components/evals3/BenchmarkPassRateChart.tsx [41-42]

 const start = new Date(minT);
-if (dense) start.setMinutes(0, 0, 0); else start.setHours(0, 0, 0, 0);
+if (dense) {
+  const h = start.getHours();
+  start.setHours(Math.floor(h / 6) * 6, 0, 0, 0);
+} else {
+  start.setHours(0, 0, 0, 0);
+}
Suggestion importance[1-10]: 7

__

Why: Valid improvement for tick alignment. The current code rounds down to the current hour, but for a 6-hour step the ticks should align to 0/6/12/18 boundaries. This produces more intuitive axis labels when data starts mid-interval (e.g., at 14:00).

Medium
Use smaller fallback pad for single-instant runs

When all runs occur at the same instant (maxT === minT), the fallback pad of one day
(86,400,000 ms) is arbitrary and can produce a misleadingly wide time axis. Consider
a smaller fallback (e.g., 1 hour) or dynamically scale based on the chart's intended
granularity to avoid visual confusion.

components/evals3/BenchmarkPassRateChart.tsx [107]

-const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 86_400_000;
+const pad = maxT - minT > 0 ? (maxT - minT) * 0.04 : 3_600_000;
Suggestion importance[1-10]: 6

__

Why: Reasonable UX improvement. When all runs occur at the same instant, the current 1-day fallback pad produces an unnecessarily wide time axis. A smaller fallback (1 hour) would be more proportional and less visually misleading while still preventing axis collapse.

Low
Validate non-negative inputs for pass rate

When passed or failed are negative (which should never happen but isn't validated),
evaluable can be negative yet pass the <= 0 guard, leading to a nonsensical negative
percentage. Add explicit validation or clamp inputs to prevent undefined behavior
from corrupted data.

lib/benchmarkRunsTable.ts [63-69]

 export function computePassRate(
   passed: number, failed: number,
 ): number | null {
+  if (passed < 0 || failed < 0) return null;
   const evaluable = passed + failed;
-  if (evaluable <= 0) return null;
+  if (evaluable === 0) return null;
   return Math.round((passed / evaluable) * 1000) / 10;
 }
Suggestion importance[1-10]: 5

__

Why: Defensive programming improvement. While negative passed/failed values should never occur in practice (they're derived from counting results), explicit validation prevents undefined behavior from corrupted data and makes the function's contract clearer.

Low
Remove default now parameter for consistency

The now parameter defaults to new Date() on every call, which can produce
inconsistent results if the function is called multiple times across a midnight
boundary during rendering. Pass a stable now from the caller or compute it once at
the component level to ensure all rows use the same reference time.

components/evals3/BenchmarkRunsTable.tsx [129-137]

-export function formatRunDate(iso: string, now: Date = new Date()): string {
+export function formatRunDate(iso: string, now: Date): string {
   const d = new Date(iso);
   if (Number.isNaN(d.getTime())) return '—';
   ...
 }
Suggestion importance[1-10]: 4

__

Why: Minor consistency improvement. The default new Date() could theoretically produce inconsistent year-display logic if called across a midnight boundary during rendering, though this is unlikely in practice. Requiring the caller to pass now makes the dependency explicit.

Low

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.06691% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.19%. Comparing base (ac025b0) to head (c4e28e0).

Files with missing lines Patch % Lines
components/evals3/BenchmarkRunsTable.tsx 67.46% 17 Missing and 10 partials ⚠️
components/evals3/BenchmarkRunsPage.tsx 71.42% 5 Missing and 7 partials ⚠️
lib/benchmarkRunsTable.ts 87.87% 7 Missing and 5 partials ⚠️
components/evals3/BenchmarkPassRateChart.tsx 82.22% 6 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #479      +/-   ##
==========================================
+ Coverage   70.09%   70.19%   +0.09%     
==========================================
  Files         413      416       +3     
  Lines       34033    34256     +223     
  Branches    10128    10212      +84     
==========================================
+ Hits        23857    24047     +190     
- Misses       7915     7937      +22     
- Partials     2261     2272      +11     
Flag Coverage Δ
e2e 48.50% <69.44%> (+0.39%) ⬆️
integration 45.58% <46.46%> (+0.09%) ⬆️
unit 77.33% <85.41%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
components/evals3/BenchmarkPassRateChart.tsx 82.22% <82.22%> (ø)
components/evals3/BenchmarkRunsPage.tsx 61.53% <71.42%> (+3.05%) ⬆️
lib/benchmarkRunsTable.ts 87.87% <87.87%> (ø)
components/evals3/BenchmarkRunsTable.tsx 67.46% <67.46%> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit 35f21e9

@goyamegh

goyamegh commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Verification

Built from this branch's head and run against a real OpenSearch storage cluster (AH_PORT=4361, run-recovery disabled so it can't touch other servers' in-flight runs). All test data is created with uniqueTestName() and deleted by id via the tracker.

Unit (npx jest tests/unit/lib/benchmarkRunsTable.test.ts tests/unit/components/evals3/BenchmarkRunsPage.test.ts tests/unit/components/evals3/BenchmarkRunsTable.test.ts)

Tests:       48 passed, 48 total

Integration (AH_PORT=4361 npx jest tests/integration/server/routes/storage/benchmarkRunsTableFields.integration.test.ts)

✓ round-trips judgeModelId / evaluatorId / testCaseSnapshots / results on embedded runs
✓ the table row derived from the real response shows Size=2, Pass %=50 (ignoring stale stats), Judge + J. Model ids
✓ preserves those fields under ?runsSize= pagination (what the page actually requests) and reports totals
✓ fields=polling (the in-flight poll request) still carries the verdict + judge fields the table re-derives on each poll
Tests:       4 passed, 4 total

E2E (PLAYWRIGHT_SKIP_WEBSERVER=1 npx playwright test tests/e2e/benchmark-runs-table.spec.ts tests/e2e/benchmarkruns-associated-evalruns.spec.ts tests/e2e/benchmark-runs-passed-count.spec.ts, then the new spec again with --repeat-each=2)

7 passed (13.1s)
10 passed (21.1s)

Scripted interaction check against a real 28-run benchmark (Playwright script, not a test — 3 agents, mixed running/cancelled/completed):

rows before filter:            27
click Agent cell on row 1  →   6 rows, pills: ["Agent: <agent A>"]
click Judge cell on row 1  →   6 rows, pills: ["Agent: <agent A>", "Judge: <evaluator>"]
legend entry aria-pressed:     1
expand row 1               →   1 heat strip rendered
click first pill (remove)  →   16 rows
Clear                      →   27 rows

Rows measure ~30px tall (were ~140px cards); the blank band between the tab strip and the list is gone (chart top sits <60px below the tabs).

CI note: the e2e-tests job on this PR fails on the same 5 specs that fail on main's own latest push CI (run 33817338957 @ 48a463a): benchmark-run-row-link ×2, benchmark-version-link-panel, evalruns-no-left-status-icon, judge-tab-whyfix. None of the specs added/changed here are among them; one new-spec test was flaky on its first attempt under worker contention (page loader bounced on a slow backend GET) and is now routed through a one-retry openRunsTab() helper.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit 6cc55d7

…me chart with click-to-filter pills

Runs tab renders a dense table (Run link · Agent · Model · Size · Pass % ·
Judge · J. Model · Date) under a per-agent pass-rate line chart. Clicking a
categorical cell or a chart legend entry toggles a filter; active filters
show as removable pills (OR within a field, AND across fields). Agent filters
dim the other chart lines instead of dropping them so the legend stays usable
as a toggle. Heat strips move behind a per-row expand chevron. Page padding,
header and row height tightened (~30px rows vs ~140px cards).

Also fixes the ~400px blank band above the runs list: the inactive Cases
TabsContent's 'flex' class out-specified the UA [hidden]{display:none} rule
and kept taking flex-1 space; data-[state=inactive]:hidden restores it.

Repairs tests/unit/components/evals3/BenchmarkRunsPage.test.ts, which had
been failing on 'useLocation is not a function' since the tabs refactor.

Tests: unit (lib + table + page), integration (benchmark GET field contract
on a real backend incl. pagination/polling), e2e (new spec + two re-targeted).

Signed-off-by: goyamegh <goyamegh@amazon.com>
(cherry picked from commit b55a6fc)
…multi-series tooltip (codex review)

- buildPassRateSeries orders series alphabetically instead of by point
  count: series index drives line colour, so busiest-first recoloured an
  agent mid-poll whenever another agent overtook it.
- Latest badge: latestRunId() = max createdAt over the version-filtered
  merged runs. filteredRuns[0] (pre-existing) missed standalone eval-run
  docs, which are appended after embedded runs.
- Chart tooltip renders every payload entry at the hovered x, not [0].
- Drop the untyped activeDot.onClick → payload.runId navigation and the
  dead allowDuplicatedCategory prop on a numeric axis.
- Pass % cell tooltip states its denominator (judged = passed+failed).

Signed-off-by: goyamegh <goyamegh@amazon.com>
(cherry picked from commit b41c6de)
…t lib/utils helpers

getJudgeModelLabel/getEvaluatorLabel exist only on main-goyamegh (from an
open PR), not on origin/main. Inline the two one-liners so the branch
applies cleanly to origin/main for the PR.

Signed-off-by: goyamegh <goyamegh@amazon.com>
(cherry picked from commit 8938ec9)
Signed-off-by: goyamegh <goyamegh@amazon.com>
…r contention

The page's loader navigates back to the benchmarks list when
GET /api/storage/benchmarks/:id fails (pre-existing behaviour); in CI the
seeded benchmark's first GET occasionally timed out under parallel workers,
which made the first attempt of one test flake (passed on retry). Route all
five tests through openRunsTab(), which retries the navigation once after a
bounce and then asserts on the table as before.

Signed-off-by: goyamegh <goyamegh@amazon.com>
(cherry picked from commit f53ea0a)
@goyamegh
goyamegh force-pushed the goyamegh/runs-table-graph-pr branch from 6cc55d7 to c4e28e0 Compare September 5, 2026 00:11
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit c4e28e0

@goyamegh

goyamegh commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up for reviewers / compose seam with #468 (goyamegh/run-actions)

This PR and #468 both touch components/evals3/BenchmarkRunsPage.tsx. In isolation both merge cleanly onto main; composed together they conflict on two import lines only:

Resolution used on the integration branch (whichever PR lands second should apply the same):

  1. Drop both Checkbox and JudgeModelSelect imports — the Runs tab is this PR's BenchmarkRunsTable + BenchmarkPassRateChart; the run form is feat(runs): complete run lifecycle actions — cancel/delete/re-run/retry with a prefilled run dialog #468's shared RunConfigDialog (with Concurrency, reused by Re-run).
  2. Keep the Evaluator import and the page-level evaluators fetch — the table's Evaluator column uses it to label rows by name (evaluatorLabel); the dialog owns its own list for the picker.
  3. Row actions stay as this PR's inline Delete/Cancel (useBenchmarkCancellationPOST /benchmarks/:id/cancel, which feat(runs): complete run lifecycle actions — cancel/delete/re-run/retry with a prefilled run dialog #468 hardened with the zombie-cancel fallback). feat(runs): complete run lifecycle actions — cancel/delete/re-run/retry with a prefilled run dialog #468 deliberately keeps its kebab (RunActionsMenu) off this page, so there is one action surface per row, not two.
  4. Cases tab (Benchmark case review: Cases/Runs tabs, master–detail with suite health, per-run heat strips #447) is untouched.

Verified on the composed tree: tsc clean; unit suites for BenchmarkRunsPage, BenchmarkRunsTable, RunConfigDialog, benchmarkRunsTable, runActions green (88/88); e2e benchmark-runs-table, benchmark-runs, benchmark-runs-passed-count, run-actions-menu green.

goyamegh added a commit to goyamegh/dashboards-traces that referenced this pull request Sep 5, 2026
…h-project#468 (kebab header) and opensearch-project#479 (runs table)

This PR was cut from origin/main and merges cleanly against it, but it
collides with two sibling PRs in the integration compose:

- components/evals3/EvalRunDetailPage.tsx, RunInspectorPage.tsx: opensearch-project#468 trims
  the lucide import lists (RotateCcw/RotateCw/GitCompare go away with the
  kebab-only header) and adds `getRunActionVisibility` right after the
  `computeRunStats` import — the same lines this PR extended with `Ban` and
  `passRateOverJudged`. Pre-align with opensearch-project#468: leave those lines untouched and
  add the new symbols on their own import lines anchored on imports neither
  PR modifies. Zero behaviour change.

- components/evals3/BenchmarkRunsPage.tsx: opensearch-project#479 replaces the card list
  (including `getRunStats` and the per-card stats strip) with the compact
  `BenchmarkRunsTable`, so this PR's "n not run" cell in that card strip has
  no home once opensearch-project#479 lands. Drop the edit here (revert the file to origin/main);
  the table's row builder (`lib/benchmarkRunsTable.ts`) should subtract
  `notRun` from `pending` as a follow-up on whichever PR lands second.
  CHANGELOG wording adjusted accordingly.

Not pre-aligned (inherently semantic, left for the integration merge):
server/routes/storage/evaluationRuns.ts cancel route — opensearch-project#468 rewrote the same
handler for the zombie-run fallback; the composed shape is "token present →
cancelRequestedAt (this PR) / no token → opensearch-project#468's direct terminal write".

Signed-off-by: goyamegh <goyamegh@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants