Skip to content

Record the Cursor event gap (R3), and wire harness + conformance into the probe - #147

Merged
jothimani-rajendran merged 2 commits into
mainfrom
claude/r3-note-and-probe-harness-wiring
Sep 13, 2026
Merged

jothimani-rajendran merged 2 commits into
mainfrom
claude/r3-note-and-probe-harness-wiring

Conversation

@jothimani-rajendran

Copy link
Copy Markdown
Collaborator

What this changes

Two commits, both closing gaps rather than adding surface.

  1. R3's note now exists in the repo (closes R3: Cursor needs a one-to-many REVERSE_EVENT_MAP once an ask-style policy ships #73) — on hj_reverse(), the function that maps a canonical event to one wire name, because that map is the limit.
  2. harness.py and conformance.py have consumers. Both shipped in Seam conformance: the harness registry, the drift definition, and a derived tier table #146 with nothing calling them. --driver harness builds a probe driver from the registry; agentseam probe conformance feeds recorded measurements to the classifier.

The practical payoff: docs/witness-skeleton.json's three commands no longer contain a hand-typed driver string. All three now read --driver harness.

Claim check

  • No capability claim is widened without a mechanism behind it — no MATRIX cell touched, no basis changed, no new recording
  • Any new/changed MATRIX row carries a verified record — none added or changed
  • Payload shapes come from a primary source — the two Cursor payloads in test_cursor_ask_dialect.py are the shapes already filed in this repo's vendor entry; the ask/deny behaviours were confirmed by executing the adapter, not read off a doc

Checks

  • pytest -q passes — 1680 passed, 4 skipped (1656 on main; +24)
  • ruff check . and ruff format --check . pass
  • Runtime path is still stdlib-only — harness gained os, shlex, subprocess, all stdlib
  • Commits are signed off

Notes for the reviewer

R3: checking it sharpened the claim. The plan said "note only, gap already resolved", which reads as closed — but the note was in neither _cursor.py nor docs/. Executing the adapter showed the runtime is already honest: ask at preToolUse returns {"permission": "deny"} with "preToolUse cannot prompt for confirmation, so this is a block", while the same decision at beforeShellExecution returns {"permission": "ask"}. So the gap sits one step earlier than the plan frames itagentseam install cursor --events pre_tool writes preToolUse and nothing else, foreclosing ask at install time, before dispatch is reached. No code changed (deny-style policies are all that ship, per R3's own terms); six tests pin the facts so the note can't rot into folklore.

A defect in my own last-round code, found by wiring it up. conformance.classify reported a single vendor as AGREED — agreeing with itself, claiming a cross-vendor check that never ran. Same family as the incapable-unanimity bug caught in #146, one row further out. The count that matters is vendors asked, not vendors capable, so one capable vendor beside an excused one is still a real VENDOR_LIMIT; a test holds that distinction so the fix can't over-correct.

Windows quoting. shlex.quote is POSIX-only and these runs happen on a Windows machine as often as not, where a POSIX-quoted argument isn't slightly wrong — it's a different command. list2cmdline on nt, shlex.quote elsewhere.

Opt-in, and the default order is asserted unchanged. --driver harness is expanded at resolution into a real command, so check_record_args, the rendered table and the report's driver field all correctly see a live driver. Whether a run counts as evidence must not shift because the registry gained a row, and a test pins the existing order (recorded if one covers this gate, else reference).

Conformance reports undecidable everywhere today, deliberately. One agent is recorded, so there is nothing to compare, and that looks like an empty feature. Two tests prove the mechanism actually decides once a second recording lands: agreement when a twin agrees, a seam gap and exit 1 when it disagrees. Nothing is fabricated to fill the table meanwhile.

Verified past the suite, since none of this is exercised by parser dispatch. A stub claude on PATH confirms the registry's argv reaches a real process with the prompt intact as a single argument, and the run reports hook_reached: False — the correct reading for a stub that never ran the hook. probe conformance, probe list, probe run --driver harness and tier-table were each dispatched through the CLI.

Unchanged and still yours: the three gaps themselves. They now need only --driver harness --agent-version <version>, and all three are --agent claude_code.


Generated by Claude Code

R3 is the last purity requirement and the only one that was listed as closed
while being absent: the plan said "note only, gap already resolved", and the
note was in neither adapters/_cursor.py nor docs/. It existed solely as issue
#73, which is not where someone shipping an ask-style policy will look.

The note now sits on hj_reverse(), the function that actually maps a canonical
event to one wire name -- that map IS the limit, so it is where the constraint
belongs.

Checked rather than quoted, and the checking sharpened it. The adapter already
handles the dialect correctly at dispatch: `ask` at preToolUse comes back
{"permission": "deny"} with "preToolUse cannot prompt for confirmation, so this
is a block", while the same decision at beforeShellExecution returns
{"permission": "ask"}. So the runtime is honest. The gap is one step earlier, on
the install side: `agentseam install cursor --events pre_tool` writes preToolUse
and nothing else, so an ask-style policy is foreclosed before dispatch is ever
reached. The plan's framing ("agentseam will need to select by decision
dialect") is right about the fix and understates where the cost lands.

No code change, per R3's own terms -- deny-style policies are all that ship and
they are unaffected. What changes is that six tests now pin the three facts the
note rests on, including that both payloads parse to the same canonical event
(which is why one name cannot pick the right gate) and that installing at
pre_tool wires the gate that cannot ask. A note nobody executes decays into
folklore; these fail the day the assumption moves.

1662 passed, 4 skipped. ruff and ruff format clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
Two modules shipped last round with nothing consuming them. harness.py knew how
to reach each vendor and nothing asked it; conformance.py could judge a
divergence and nothing fed it one. Both are now wired to the probe, which is
where a witnessed run actually happens.

**The driver comes from the registry.** `--driver harness` builds the invocation
from harness.json instead of the operator typing it, so the three open coverage
gaps lose their hand-written driver string -- docs/witness-skeleton.json now
carries `--driver harness` on all three commands, pinned by a test that resolves
it through the registry rather than matching the literal, so deleting
claude_code's row breaks the test instead of leaving three commands that resolve
to nothing.

Quoted per platform, not POSIX-only: these runs happen on Windows as often as
not, and shlex.quote there does not produce a slightly-wrong command line, it
produces a different one. list2cmdline on nt, shlex.quote elsewhere. The prompt
slot stays unquoted because drive_real substitutes an already-quoted JSON string
for it -- a contract now exercised end to end rather than assumed.

Opt-in by name, and expanded at resolution. The default order (recorded if one
covers this gate, else reference) is untouched and asserted so: whether a run
counts as evidence must not shift because a registry gained a row. Expanding the
token into a real command means check_record_args, the table and the report's
driver field all see a live driver, which it is -- no special-casing in three
places to stop it reading as non-live.

**Conformance runs over the recordings.** `agentseam probe conformance` replays
each recorded trial through the real classifier (no process launched) and asks
what a difference between agents means. With one agent recorded it reports
undecidable everywhere, which is the honest answer and looks like an empty
result; two tests prove the mechanism decides once a second recording lands --
agreement when the twin agrees, a seam gap and exit 1 when it does not.

Wiring it up found a defect in conformance.py: a single vendor was reported
AGREED, agreeing with itself, claiming a cross-vendor check that never ran. Same
family as the incapable-unanimity bug caught last round, one row further out.
The count that matters is vendors asked, not vendors capable -- one capable
vendor beside an excused one is still a real vendor-limit finding, and a test
holds that distinction.

Verified beyond the suite, since none of this is exercised by parser dispatch: a
stub `claude` on PATH confirms the registry's argv reaches a real process with
the prompt intact as one argument, and the run reports hook_reached=False, the
correct reading for a stub that never ran the hook.

1680 passed, 4 skipped. ruff and ruff format clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
@jothimani-rajendran
jothimani-rajendran marked this pull request as ready for review September 13, 2026 23:14
@jothimani-rajendran
jothimani-rajendran merged commit 3bef784 into main Sep 13, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

R3: Cursor needs a one-to-many REVERSE_EVENT_MAP once an ask-style policy ships

2 participants