Repository navigation
Workbench: live measurements leave no versioned trace #459
Description
Split out of #428, which is being closed — its core ask is done, this is the adjacent point that was never in scope for it.
What's left
workbench/runs/ and workbench/reports/ are gitignored, and only three baselines are committed (consent.json, describe-project.json, wizard-enhance.json). Every live measurement made against this bench — including the ones behind #431, #432, #440, #452, #453, #454 and #456 — leaves no trace anyone else can replay without re-running it themselves, on their own key, against whatever the provider resolves that day.
Concretely, that means:
- A cassette recorded during a PR review (e.g.
workbench/cassettes/judge-migration-*.json) replays offline for whoever has the file, but nothing forces it to reachmain— it's a byproduct of one session, not a tracked measurement. - The three real-take scenarios (
real-wizard-enhance,real-zooms,real-zoom-grounding) have never been judged at all, because the take itself (workbench/fixtures/) is gitignored and absent from every clone. See removeModifier deletes a whole pill but reports removing one id #457's sibling problem andworkbench/README.md§ "La prise réelle" for what that costs on a fresh checkout (44 L0 tests fail on the missing fixture, silently, since CI never runs the bench). - A baseline's
behaviour/dslnumbers, once written, are never re-verified against anything — nothing reads them back (baseline.tsonly compares check ids), so they drift into being an unlabelled archive, which is exactly what happened towizard-enhance.jsonbefore fix(workbench): retire beh.sandbox from the baseline, confirmed at n=10 #453 said so explicitly.
First step, as noted when this was flagged
Freeze the real-take scenario baselines (--update-baseline, verified by hand rather than trusted), then capture the provider's usage block in the recording proxy (workbench/lib/cassette.ts) — the token count every context-size argument in this codebase wants is already in the response the recorder sees pass through, uncaptured.
Also worth deciding here, not in #428
Whether cassettes from a judged PR review should be committed as a matter of course (the way judge-migration-*.json were, by convention rather than by rule) or whether that's noise the repo doesn't want. workbench/README.md doesn't currently say either way.
Activity
auberginewly commentedon Sep 26, 2026
Hey! I'd love to help move this forward!
I'm currently working on an internal Coding Agent during my internship, and I'm particularly interested in agent evaluation and reliability.
I noticed that the original real-take baselines are still missing. I'd be happy to take ownership of this part, including building reproducible evaluation scenarios, preparing test fixtures, and validating the results.
Are the original recording fixtures available to contributors, or would it make more sense to create a new set?
I'd love to contribute more regularly to OpenScreen, especially on the Agent side. Happy to take on more work in this area over time!
EtienneLescot commentedon Oct 24, 2026
Hi @auberginewly, sorry for the slow reply, and thanks for offering! The original take stays local and isn't distributed, so a new, small fixture set that can be committed is the better route: it also fixes fresh clones missing the fixture. Good first steps: freeze the real-take scenario baselines (checked by hand), then capture the provider's
usageblock inworkbench/lib/cassette.ts. Happy to review, ask anything here.auberginewly commentedon Oct 27, 2026
Thanks for clarifying, @EtienneLescot! I'll use a new, small fixture set that can be committed, with distinct scenario IDs and separately checked annotations, while keeping the original private take and its baselines explicitly unavailable.
I also checked the merged #653: the cassette usage capture and versioned measurement/replay workflow are already there, so I'll build on those rather than duplicate them. The remaining work is the public material and scenarios, recorded measurements, and reviewed baselines. Thanks for the direction!
auberginewly commentedon Oct 27, 2026
Thanks again for the guidance, @EtienneLescot. I've opened draft #1403 with a small publicly committable local recording, its project/cursor sidecar, capture source and licensing, and three new scenario IDs. It reuses #653's measurement workflow, with an explicit
public-recordedregistration rather than labelling a recording synthetic. The original private take andreal-*failures are preserved.The scoped offline checks and measurement lifecycle controls pass, and the PR includes a candid independent agent review record. This fixture is a silent, scripted browser capture, so it does not yet cover narrated enhancement or speech preservation. Real provider measurements are also still pending local
.env.workbenchconfiguration; there are no claimed provider usage numbers, judged model results or approved baselines yet. I'll keep the PR in draft while those gaps remain, and I'm not treating this as completion of #459. Thank you for offering to review.