Seam conformance: the harness registry, the drift definition, and a derived tier table - #146
Merged
jothimani-rajendran merged 3 commits intoSep 13, 2026
Conversation
Owner decision, 2026-09-13: vendor knowledge belongs in the seam, not in a prototype. prototypes/a11y-guard/scenarios/scenarios.json holds an `agents` map that is a vendor adapter registry in all but name -- argv per vendor, the model flag plus the position it has to go in, what voids a run, how Codex gates its sandbox on trust recorded outside the workspace, and that a file the agent creates on Windows is owned by an account the harness cannot read. None of that is inferable and all of it was paid for by losing runs. Eleven Codex runs were voided by a throwaway CODEX_HOME carrying no trust entries; `--full-auto` stopped existing between two minor versions. Left in a prototype, the next pack rediscovers each at the same price. Generated from that file rather than retyped, so nothing was lost in transcription, and keyed by canonical matrix ids -- a row whose agent the matrix does not know is now a test failure rather than a lookup that quietly misses. The module's invariant is a refusal: an agent with no recorded row gets NoHarnessError, never a guessed command line. A guess is the failure mode worth designing against because its output is indistinguishable from a real result. The same discipline covers the hooked arm -- codex_cli has no established one, and asking for it raises with the recorded reason instead of silently falling back to the unhooked argv and measuring the wrong thing. void_markers is that rule aimed at output: it names evidence the agent was PREVENTED from working, which is not an agent that looked and chose to do nothing. The first measured nothing; the second is the data point. The caller applies the gate, since it only discriminates for a run that changed no files. 17 tests, including that no invocation is hardcoded in the module and that a prompt containing braces survives (placeholders are replaced, never formatted). ruff and ruff format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
The 2026-09-13 decision says a policy is tested against the seam and the seam against the vendors, once -- and that if drift remains, agentseam is half cooked. That rule could be quoted but not enforced, because nothing said which differences count. Not all of them do: copilot failing open where others fail closed is a true property of copilot, and the matrix says so out loud. The line drawn here: a policy's verdict must be identical across every vendor that CAN reach it, and where a vendor cannot, the shortfall is reported rather than papered over. A verdict differing between two vendors the matrix says are equally able has no explanation in the world, so it is this layer leaking a dialect into a policy's result. SEAM_GAP is that call, and is_seam_gap() is the one-line form a CI gate can hold. Named divergence, not drift, on purpose: staleness.DRIFTED already means a vendor shipped a new version since a row was recorded. Same English word, unrelated question, and this track has lost time to that kind of collision before. A test asserts the word stays out of the code here. Two refusals carry the honesty. An unrecorded vendor is never excused -- treating it as "cannot enforce" would let any divergence be waved through by the absence of evidence, which is the papering-over the rule forbids; capable() raises where it would otherwise have to answer False. And capability is settled BEFORE agreement: two vendors that cannot enforce will always return the same thing, and an earlier revision of this file reported that as AGREED -- an untested policy reading as a passing one. Caught by running the cases rather than by reading the diff, and the regression test says why it must stay in that order. 14 tests. ruff and ruff format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Third obligation of the 2026-09-13 decision: if the seam owns vendor conformance, the per-agent enforcement table is its output, not a thing kept in step by hand in each consumer's instructions. A hand-written copy drifts silently and in the direction that flatters -- nobody notices a table still claiming last month's tier, and a table is exactly where a reader goes to avoid reading the code. The grade column is why this is worth building rather than tidying: it is capped by basis, so a row resting on vendor docs cannot print `enforced` however confident its cell reads. A hand-written table has no such mechanism. Rendering the derived table against the hand-written one in the chock-family instructions disagrees on every row they share, which is the argument stated as a result. What it deliberately will not emit is the consumer's own column. T3, `hook + CI` and `CI only` are chock's vocabulary for how chock wires a control; agentseam does not know them and must not learn them. It renders the columns it owns and a consumer joins its wiring column on -- asking for a column this table does not own is a refusal, not a blank cell. A test asserts no consumer vocabulary reaches either the output or the module body. An agent with no hook surface prints n/a rather than a weak fail mode: there is nothing there to fail, and saying so keeps the absence visible. cli.py was exactly at the 300-line review budget, so the read-only verbs moved to cli_report.py. Split by activity, which is the remedy the budget test names: these three report on evidence, everything left writes to a machine, and a verb that only prints is safe to run anywhere. No behaviour changed -- matrix, matrix --evidence and doctor were each re-run through the CLI after the move, since the suite does not exercise parser dispatch. 10 tests for the table. 1656 passed, 4 skipped. ruff and ruff format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The three obligations the 2026-09-13 owner decision ("policies test the seam, not the vendors") created, each as its own commit and none of them needing a vendor token to build:
harness.py+data/harness.json— the vendor registry the a11y pack paid for, moved out ofprototypes/a11y-guard/scenarios/scenarios.json: argv per vendor, the model flag and the position it must go in, what voids a run, Codex's trust gating, and Windows file ownership.conformance.py— the definition of the difference that condemns this layer, so "then agentseam is half cooked" is testable rather than quotable.tier_table.py+agentseam tier-table— the enforcement table derived from the matrix instead of hand-maintained in each consumer's instructions.No matrix row, capability or basis changed. Nothing here claims a new measurement.
Claim check
MATRIXcell is touched;tier_tableonly readsenforcement_level(), which is already basis-cappedMATRIXrow carries averifiedrecord — no rows added or changeddata/harness.jsonwas generated fromprototypes/a11y-guard/scenarios/scenarios.jsonrather than retyped, so the recorded prose (Codex trust inheritance, the--full-autorename between 0.150.1 and 0.154.0, Windows file ownership) arrives verbatim from the runs that established it. Agent ids were remapped to canonical matrix ids (codex→codex_cli,claude→claude_code), and a test asserts every harness row is a known matrix row.Checks
pytest -qpasses — 1656 passed, 4 skipped (1615 on main; +41 from the three new test files)ruff check .andruff format --check .pass — both clean, fixed per-rule rather than by a blanket--fixjson/pathlibvia the existing_data.loadNotes for the reviewer
Two defects I introduced and caught, both by running cases rather than reading the diff:
conformance.classifyshort-circuited on agreement before checking capability, so two vendors that cannot enforce — which will always return the same thing — reported asAGREED. An untested policy reading as a passing one is precisely the papering-over the module exists to forbid. Capability is now settled first, andtest_unanimity_among_vendors_that_cannot_enforce_is_not_agreementpins the order.test_the_word_drift_is_not_reused_for_this_conceptwas written as a fragile occurrence-count and asserted the wrong arithmetic. Rewritten to state the intent directly: the word may appear in the docstring, never in the code.Worth your judgement — the derived table disagrees with the hand-written one on every row they share. Against the four rows in the chock-family instructions:
claude_codederivesfail_mode: open/basis: live-runwhere the table says fail-closed / vendor-docs;cursorderivesconfigurable/live-run-partialagainst fail-closed / vendor-docs;devinderivesblock+rewrite/openagainstn/a; andcopilotderivesunadaptedwith no hook surface at all against T3 / hook + CI. Theclaude_codefail-open reading matches the witnessed run. But the two tables'Fail modecolumns may not be asking the same question — the hand-written one is captioned "where each agent reading this file is actually stopped", which could mean the control fails closed rather than the vendor's hook. Worth settling what that column means before anything regenerates from this.Purity.
tier_tablerefuses to render a column it does not own, soT3/hook + CI/CI onlycannot leak into the engine; a test asserts no consumer vocabulary reaches the output or the module body.The
cli.pysplit.cli.pysat at exactly the 300-line budget, so the read-only verbs moved tocli_report.py— split by activity, the remedy the budget test names. No behaviour changed, and since the suite does not exercise parser dispatch I re-ranmatrix,matrix --evidence,doctorand bothtier-tableforms through the CLI by hand.Still needs your machine, unchanged by this PR: the three
witness-skeleton.jsongaps (pre_tool/escalate,prompt_submit,stop) — all three are--agent claude_code, so no other vendor's tokens are involved.Generated by Claude Code