Record the Cursor event gap (R3), and wire harness + conformance into the probe - #147
Merged
jothimani-rajendran merged 2 commits intoSep 13, 2026
Merged
Conversation
R3 is the last purity requirement and the only one that was listed as closed while being absent: the plan said "note only, gap already resolved", and the note was in neither adapters/_cursor.py nor docs/. It existed solely as issue #73, which is not where someone shipping an ask-style policy will look. The note now sits on hj_reverse(), the function that actually maps a canonical event to one wire name -- that map IS the limit, so it is where the constraint belongs. Checked rather than quoted, and the checking sharpened it. The adapter already handles the dialect correctly at dispatch: `ask` at preToolUse comes back {"permission": "deny"} with "preToolUse cannot prompt for confirmation, so this is a block", while the same decision at beforeShellExecution returns {"permission": "ask"}. So the runtime is honest. The gap is one step earlier, on the install side: `agentseam install cursor --events pre_tool` writes preToolUse and nothing else, so an ask-style policy is foreclosed before dispatch is ever reached. The plan's framing ("agentseam will need to select by decision dialect") is right about the fix and understates where the cost lands. No code change, per R3's own terms -- deny-style policies are all that ship and they are unaffected. What changes is that six tests now pin the three facts the note rests on, including that both payloads parse to the same canonical event (which is why one name cannot pick the right gate) and that installing at pre_tool wires the gate that cannot ask. A note nobody executes decays into folklore; these fail the day the assumption moves. 1662 passed, 4 skipped. ruff and ruff format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Two modules shipped last round with nothing consuming them. harness.py knew how to reach each vendor and nothing asked it; conformance.py could judge a divergence and nothing fed it one. Both are now wired to the probe, which is where a witnessed run actually happens. **The driver comes from the registry.** `--driver harness` builds the invocation from harness.json instead of the operator typing it, so the three open coverage gaps lose their hand-written driver string -- docs/witness-skeleton.json now carries `--driver harness` on all three commands, pinned by a test that resolves it through the registry rather than matching the literal, so deleting claude_code's row breaks the test instead of leaving three commands that resolve to nothing. Quoted per platform, not POSIX-only: these runs happen on Windows as often as not, and shlex.quote there does not produce a slightly-wrong command line, it produces a different one. list2cmdline on nt, shlex.quote elsewhere. The prompt slot stays unquoted because drive_real substitutes an already-quoted JSON string for it -- a contract now exercised end to end rather than assumed. Opt-in by name, and expanded at resolution. The default order (recorded if one covers this gate, else reference) is untouched and asserted so: whether a run counts as evidence must not shift because a registry gained a row. Expanding the token into a real command means check_record_args, the table and the report's driver field all see a live driver, which it is -- no special-casing in three places to stop it reading as non-live. **Conformance runs over the recordings.** `agentseam probe conformance` replays each recorded trial through the real classifier (no process launched) and asks what a difference between agents means. With one agent recorded it reports undecidable everywhere, which is the honest answer and looks like an empty result; two tests prove the mechanism decides once a second recording lands -- agreement when the twin agrees, a seam gap and exit 1 when it does not. Wiring it up found a defect in conformance.py: a single vendor was reported AGREED, agreeing with itself, claiming a cross-vendor check that never ran. Same family as the incapable-unanimity bug caught last round, one row further out. The count that matters is vendors asked, not vendors capable -- one capable vendor beside an excused one is still a real vendor-limit finding, and a test holds that distinction. Verified beyond the suite, since none of this is exercised by parser dispatch: a stub `claude` on PATH confirms the registry's argv reaches a real process with the prompt intact as one argument, and the run reports hook_reached=False, the correct reading for a stub that never ran the hook. 1680 passed, 4 skipped. ruff and ruff format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
jothimani-rajendran
marked this pull request as ready for review
September 13, 2026 23:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Two commits, both closing gaps rather than adding surface.
hj_reverse(), the function that maps a canonical event to one wire name, because that map is the limit.harness.pyandconformance.pyhave consumers. Both shipped in Seam conformance: the harness registry, the drift definition, and a derived tier table #146 with nothing calling them.--driver harnessbuilds a probe driver from the registry;agentseam probe conformancefeeds recorded measurements to the classifier.The practical payoff:
docs/witness-skeleton.json's three commands no longer contain a hand-typed driver string. All three now read--driver harness.Claim check
MATRIXcell touched, no basis changed, no new recordingMATRIXrow carries averifiedrecord — none added or changedtest_cursor_ask_dialect.pyare the shapes already filed in this repo's vendor entry; theask/denybehaviours were confirmed by executing the adapter, not read off a docChecks
pytest -qpasses — 1680 passed, 4 skipped (1656 on main; +24)ruff check .andruff format --check .passharnessgainedos,shlex,subprocess, all stdlibNotes for the reviewer
R3: checking it sharpened the claim. The plan said "note only, gap already resolved", which reads as closed — but the note was in neither
_cursor.pynordocs/. Executing the adapter showed the runtime is already honest:askatpreToolUsereturns{"permission": "deny"}with "preToolUse cannot prompt for confirmation, so this is a block", while the same decision atbeforeShellExecutionreturns{"permission": "ask"}. So the gap sits one step earlier than the plan frames it —agentseam install cursor --events pre_toolwritespreToolUseand nothing else, foreclosingaskat install time, before dispatch is reached. No code changed (deny-style policies are all that ship, per R3's own terms); six tests pin the facts so the note can't rot into folklore.A defect in my own last-round code, found by wiring it up.
conformance.classifyreported a single vendor asAGREED— agreeing with itself, claiming a cross-vendor check that never ran. Same family as the incapable-unanimity bug caught in #146, one row further out. The count that matters is vendors asked, not vendors capable, so one capable vendor beside an excused one is still a realVENDOR_LIMIT; a test holds that distinction so the fix can't over-correct.Windows quoting.
shlex.quoteis POSIX-only and these runs happen on a Windows machine as often as not, where a POSIX-quoted argument isn't slightly wrong — it's a different command.list2cmdlineonnt,shlex.quoteelsewhere.Opt-in, and the default order is asserted unchanged.
--driver harnessis expanded at resolution into a real command, socheck_record_args, the rendered table and the report'sdriverfield all correctly see a live driver. Whether a run counts as evidence must not shift because the registry gained a row, and a test pins the existing order (recordedif one covers this gate, elsereference).Conformance reports
undecidableeverywhere today, deliberately. One agent is recorded, so there is nothing to compare, and that looks like an empty feature. Two tests prove the mechanism actually decides once a second recording lands: agreement when a twin agrees, a seam gap and exit 1 when it disagrees. Nothing is fabricated to fill the table meanwhile.Verified past the suite, since none of this is exercised by parser dispatch. A stub
claudeonPATHconfirms the registry's argv reaches a real process with the prompt intact as a single argument, and the run reportshook_reached: False— the correct reading for a stub that never ran the hook.probe conformance,probe list,probe run --driver harnessandtier-tablewere each dispatched through the CLI.Unchanged and still yours: the three gaps themselves. They now need only
--driver harness --agent-version <version>, and all three are--agent claude_code.Generated by Claude Code