integration: the seven reviewed waves of 2026-09-07 (#114 #115 #116 #118 #119 #120 #117) - #121
Merged
Merged
Conversation
Finding raw[14].findings[12] of the vendor-truth review, reproduced by execution: both entries mapped the vendor's post-compaction event onto pre_compact beside their real PreCompact, so a handler that exists to snapshot context before compaction also fired after it had already happened. Devin's identical defect was closed in PR #43; the "unknown" treatment Kimi's four aliases got in PR #59 keeps claims() identifying the payload, so only the relabelling stops. wire_events already pinned pre_compact to PreCompact on both, so install output is unchanged. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
experiment_report.as_report() now reads `event` off the trial results (every result already carries one) instead of leaving it implicit in the file the report happens to be pasted under. evidence_report.py accepts `event` as an optional field, validated against contract.EVENTS when present, and carries it through to_evidence()/diff_against() so a maintainer merging a submission can see which gate it measured rather than assuming the row's own. Existing reports with no `event` stay valid. The evidence-report.yml issue template gains a matching optional field. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Add trial `escalate` (tools/experiment_probe.py): the probe answers with Decision.escalate() rendered through adapter.respond(), so Cursor's reply spells it "ask" and an agent whose gate does not honour escalate gets its own degraded-block dialect, exactly like every other trial's answer. The new measured field `escalate_means` needs a third value neither `silence_means` nor `unknown_verb_means` has to express: `prompted`, when the run ended waiting on an answer nobody gave, distinct from a definite `refusal-or-error`. The sentinel alone cannot tell those apart -- both leave it untouched -- so classification also reads the driver's own outcome (non-zero exit or timeout, sentinel untouched, reads as `prompted`); split into tools/experiment_escalate.py, which carries the exact rule in its docstring, to keep experiment.py under the line budget. tools/experiment.py's own real-driver invocation moves to tools/experiment_driver.py for the same reason, gaining a caught TimeoutExpired so a stalled headless run is observed rather than propagated as a crash. tools/reference_agent.py now raises Undocumented for PreToolUse's `permissionDecision: "ask"`: the value is documented, but what a headless run does next with nobody there to answer is not, so a guess here would launder that silence into evidence -- the same discipline the unknown-verb trial already gets. escalate_means joins matrix_terms.OPTIONAL_CLAIM_FIELDS with the same "absence is not a claim" rule as W53's fields. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
…w basis claim_basis() fell back to a row's own live-run-partial basis for any event the row claims, whether or not that event was ever watched -- and W53's mechanical seeding gave every cell, watched or not, a per-claim record copying the row basis verbatim, so the fallback path was not even needed to trigger it. A live-run-partial row could therefore back `enforced` at an event its own `verified.observed` never names, which is exactly the "partial" word's whole point to prevent. claim_basis() now checks `observed` itself: when the resolved basis is live-run-partial and the event is not in it, the effective basis is the row's new optional `verified.fallback_basis` (what the rest of the row rests on), defaulting to vendor-docs when the row does not say one. Set from each row's own method text: codex_cli and vscode_copilot say source (vendor-source); cursor says vendor hooks documentation (vendor-docs). Recomputed every (agent, event) pair's basis under the fix: 11 change (the unobserved events on these three rows, all now vendor-source or vendor-docs instead of live-run-partial), all detailed in the PR body. None of the 91 claimed pairs' actual *grade* moves -- none of those 11 cells asserts a fail-closed claim, so every one was already sitting at or below its new, lower ceiling. This closes the gap defensively, same shape as W53's original cap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Finding raw[8].findings[8] of the vendor-truth review, reproduced by execution. devin's accept_names claimed PermissionRequest before any marker check, on the ground that Claude Code never sends that name -- but Kimi Code does, so a real Kimi PermissionRequest was claimed by two adapters, detect() returned None, and handle() allowed it with no Event at all. Not even the observation value of the mapping survived, which contradicts the recorded rule that a positive self-identification beats a shared event name. claims.reject_client_types is checked ahead of accept_names and lists the one recorded collision. A Devin PermissionRequest, which carries no client_type, is claimed exactly as before -- pinned by a test that must keep passing. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finding raw[8].findings[6] of the vendor-truth review, reproduced by execution. claims() required the wire event name to be in `events`, so a payload carrying client_type=kimi_code_cli and a new or unmapped event name was claimed by no adapter: handle() returned event=None rather than the UNKNOWN Event the contract documents as the whole point of that pathway. Vendor drift on Kimi was invisible to a caller logging UNKNOWN events -- the one thing that pathway exists to make visible. claims.accept_any_name is opt-in and set on the one entry whose client_types cannot be null, so the relaxation rests on a positive self-identification rather than on a guess. A payload without client_type is still not claimed, and the decision is unchanged: an UNKNOWN event allows. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finding raw[5].findings[8] of the vendor-truth review, reproduced by execution: a rewrite degraded at UserPromptSubmit or Stop was refused with "Devin cannot modify a tool call", where no tool call was ever involved -- pointing an operator debugging why prompt sanitisation blocks at tool plumbing. This string is the one surface an end user reads. The note becomes the engine's template form and names the vendor event, matching cursor's per-gate phrasing. Four frozen wire bytes move (rewrite and rewrite-without-input at prompt_submit and stop); the PreToolUse and PermissionRequest bodies are byte-identical, and the reason default loses its "before it can run" tail for the same reason. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed path test test_a_user_scoped_config_path_is_not_nested_under_the_repo set HOME to tmp_path itself and then asserted the config landed in tmp_path/.junie. Only posixpath.expanduser reads HOME. ntpath reads USERPROFILE, then HOMEDRIVE + HOMEPATH, and conftest's autouse isolated_home fixture already points all four at a `home` directory beneath tmp_path -- so on Windows the install correctly wrote its config under the fixture's home while the assertion looked one level up, in tmp_path itself. The fixture is the platform-independent answer and the test now takes it as a parameter instead of re-deriving home from an env var that three quarters of the supported platforms ignore. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… test test_install_never_destroys_a_config_it_cannot_parse seeded a config at tmp_path/.junie/config.json after setting HOME to tmp_path, then called install() and checked the user's settings survived. On Windows expanduser resolved ~ to the fixture's tmp_path/home, so install never saw the seeded file at all: it created a fresh config elsewhere, and every assertion here was reading a file nothing had touched. The BOM case passed vacuously and the unparseable cases raised JSONDecodeError from the test's own json.loads rather than ConfigUnreadableError from install. Home now comes from the isolated_home fixture, which sets every variable expanduser actually consults on both platform families. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… query test test_a_query_never_raises_on_an_unparseable_config wrote a corrupt config under a HOME it had set by hand, then asserted installed() returns False and uninstall() raises ConfigUnreadableError. On Windows the corrupt file was not on the path either call reads, so installed() returned False because nothing was there -- the right answer for the wrong reason -- and uninstall() found no file to fail on, which is why the test reported DID NOT RAISE. Home now comes from the isolated_home fixture. All three install tests that seeded a user-scoped config by hand are now consistent with it. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…all test kimi_code's CONFIG_PATH starts with ~/, so its config is user-scoped by design -- there is a test in this same file asserting exactly that. The install test still seeded the file at tmp_path/.kimi-code/config.toml after setting HOME by hand, which resolves nowhere on Windows. The result was the sharpest form of the failure: install() wrote a correct config under the fixture's home, the test read the untouched original it had seeded, and reported that guard.py was missing from a file install had never been asked to write. Home now comes from the isolated_home fixture. tmp_path stays as the repo root argument, which is a different thing and still belongs there. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…stall test test_reinstalling_replaces_our_block_rather_than_stacking_them read the config back from tmp_path/.kimi-code/config.toml after two installs. Both installs wrote to the home expanduser actually resolves, so on Windows the read raised FileNotFoundError before any assertion about stacked blocks ran. Home now comes from the isolated_home fixture, so the file the test reads is the file the installs wrote. `Path` is still used elsewhere in the module, so the import stays. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generated pages carry em dashes. Both readers that compare a committed page against a freshly built one opened it with no encoding argument, so on a cp1252 host every em dash decoded to three characters that could never match. test_committed_pages_match_what_the_library_produces reported all twelve pages stale on Windows -- with no way to make it pass, because regenerating wrote the same bytes it had just failed to read back. Both readers now say encoding="utf-8": the test's, extracted into a _committed() helper that can carry the reason, and generate.py's --check path, which the pre-commit hook and the examples CI job both run and which had the identical latent bug. The writer says encoding="utf-8", newline="\n" for the same reason from the other side: a regeneration on Windows was writing CRLF into files the repo stores with LF, so every page showed as modified until git normalised it away. Same assumption, same fix, and it keeps a Windows regeneration byte-identical to a CI one. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_hook_is_executable read HOOK.stat().st_mode & 0o111. Windows has no POSIX execute bit, so that is 0 for every file in the tree -- including this one, which git records as 100755 and checks out executable on Linux and macOS. The test could not pass on Windows and told the truth about nothing there. It now reads the mode git records for the path. That is the mode a clone gets, which is what decides whether the hook runs, and it is the same answer on every platform. A hook that is not tracked at all now fails with its own message rather than an index error. Verified real rather than vacuous: git currently records 100755 for .githooks/pre-commit, and the assertion names the mode it found when it is anything else. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ssing test_a_query_never_raises_on_an_undecodable_toml_config carried the same hand-set HOME as the three install tests fixed earlier on this branch, but it did not fail on Windows. That is the worse outcome: installed() returned False because the utf-16 file was not on the path it reads, not because the TOML branch had decoded it safely. The test asserted its own setup. Not one of the seven reported failures. It is the same defect in the same file, found while fixing them, and it was reporting a pass for a branch it never entered. With home taken from the fixture, the file is on the path installed() reads: verified by writing a valid installed config to that path and watching installed() return True, then overwriting it with the utf-16 body and watching it return False without raising. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
data/recordings/<agent>@<version>.json holds the sentinel and hook-invocation counts a real agent produced per trial, one immutable file per (agent, version). agentseam.recordings is the package-side reader; the dev-only writer lands with the recorded driver in the next commit. Seeds claude_code@2.1.263 from the witnessed pre_tool run already on org-plan (plan/spine-a/evidence/). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
tools/recorded_driver.py is the dev-only writer and replay driver: --record freezes a real-agent run into data/recordings/, refusing the reference and recorded drivers themselves (the same live-evidence invariant evidence_report.py already enforces on a submitted report). --driver recorded (default once a recording exists for --agent) replays a trial through experiment._classify with no process launched, so the recorded table reproduces in under a second instead of a subprocess round trip. evidence_report.py gains recorded_version and rejects a recorded-driver report whose version exceeds the recording it actually replayed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Finding raw[2].findings[7] of the vendor-truth review, reproduced by execution. parse() infers afterFileEdit from an edits[] list; respond() re-derived the event from event.tool, which parse() fills from tool_name when the payload carries one, and so fell through to the entry's default beforeShellExecution gate. An unnamed edits payload with a tool_name therefore parsed as file_changed and was answered with a permission verdict at an event that reads no output -- an already-landed write reported as prevented. respond() now calls cursor_wire() itself. event.tool is read only for an Event carrying no payload, where parse() left the inferred name there and there is nothing to re-infer from. No frozen wire output moves: every golden scenario names its own event. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The six pre_tool per-claim records repeated the same live-run prose six times; each now points test at data/recordings/claude_code@2.1.263.json instead, so the basis chain is claim -> recording -> live run, version-pinned. Same basis/date/version, just no longer copy-pasted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
…nger A drift issue is a work order for whoever has the agent installed, not a note to the maintainer: watch_versions.py --open-issues now opens one issue per drifted agent instead of one combined issue, titled "evidence: <agent> <version> -- re-witness wanted", labelled evidence and help wanted (created if absent), and skips an agent that already has one open. The body carries the exact --record commands for the gates the kit can actually gate at, what running them produces, and both ways to submit the result. A covered gate now compares against its recording's own version ahead of the per-claim override. CONTRIBUTING.md points at the same steps; agentseam matrix --evidence shows the recorded version beside the row's. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
… behind each Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Finding raw[2].findings[4] of the vendor-truth review, reproduced by execution. Decision.rewrite(None, ...) at preToolUse was refused with "input requires modification, which this gate cannot express" -- untrue at the one Cursor gate that can, and the adapter's headline capability. The handler had simply supplied no replacement, so an operator reading that message files a capability bug against the wrong layer. The engine already separates the two cases with transform_missing_input; _cursor.py now consults it and keeps the "cannot express" wording for the gates where it is true. Pulling the reason out of _gate_payload into _refusal_reason is what makes that branch expressible. The deny itself, which is the safe outcome, is unchanged; one frozen wire byte moves. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finding raw[2].findings[6] of the vendor-truth review, both halves reproduced by execution. At beforeSubmitPrompt user_message was attached whenever the decision carried a reason, whatever the outcome, so a handler annotating its allows for its own audit trail put that text in Cursor's UI on every submitted prompt. The permission-gate branch has always attached messages only when not allowing. The same branch also skipped the degradation note every other gate adds, so a prompt blocked because a rewrite could not be expressed there was shown only the handler's original reason -- the mis-story _because() exists to prevent. It now shares _refusal_reason with the gate branch. The shared escalate_from_transform note says "cannot modify the input" rather than "cannot modify a tool call": a prompt gate modifies a prompt. No permission-gate scenario emitted that note, so the four frozen wire bytes that move are all at prompt_submit. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Finding raw[8].findings[4] of the vendor-truth review, reproduced by execution. _toml_value escaped only backslash and double quote, so a command or matcher carrying a newline ended the TOML line and spilled the rest into the [[hooks]] table as extra bare keys -- against a vendor rule this repository records itself: four fields only, a fifth makes the whole file fail to load. The rendered block does not parse, and neither does a user's own [model] section above it, so every hook in the file stops firing on a vendor that fails open. Silent, and total. _toml_value now emits the full basic-string escape set with \uXXXX for every other C0 control and U+007F. The block is pinned by a round-trip through tomllib where it exists, and unconditionally by a line count derived from the rule's own field count -- the newline bug's signature. No output moves for a control-character-free command, which is every committed example and fixture. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
claude_code@2.1.263 records pre_tool only. resolve_driver() defaulted to 'recorded' whenever the agent had any recording, so a run at stop or prompt_submit died on NoRecording instead of falling back to the reference. has_recording() now takes the event (and version) and resolve_driver() passes them through; the CLI passes --event and --agent-version. Test covers the recorded gate, the two unrecorded gates, an agent with no recording, and an explicit --driver override. Also: the recording schema's top-level description was a JSON array, which the repo's own validator accepted but the 2020-12 meta-schema rejects; it is one string now. Found in review of #117. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
… into integration/2026-09-07-seven-waves
…tion/2026-09-07-seven-waves
…egration/2026-09-07-seven-waves
… integration/2026-09-07-seven-waves
…ation/2026-09-07-seven-waves
…s the row's version from data Seven reviewed branches merged in review order (#114, #115, #116, #118, #119, #120, #117). CHANGELOG conflicts resolved by keeping every entry; the two code conflicts between W55 (report `event`) and W57 (recorded driver) resolved by keeping both optional fields and both checks, the pair moved into one _check_provenance() helper so validate() stays under the complexity ceiling. W57's `matrix --evidence` test hard-coded 2.1.247 for the row's version; #114 moved the row to 2.1.263, so the test now reads the version from the data. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
jothimani-rajendran
marked this pull request as ready for review
September 8, 2026 00:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
One merge of seven reviewed branches, in review order, so the owner merges once instead of resolving the same CHANGELOG conflict seven times. Merging this closes #114, #115, #116, #118, #119, #120 and #117 as merged content; nothing in those branches is altered except the reconciliation below.
--record, recorded driver, contributor drift issuesReconciliation commits (the only new code)
evidence_report.py: both optional fields (event,recorded_version) and both checks kept, moved into one_check_provenance()helper sovalidate()stays under the complexity ceiling.tools/experiment_report.py:as_reportbuilds the recorded or live shape and then setseventon either.tests/test_cli.py: W57'smatrix --evidencetest hard-coded the row version 2.1.247, which matrix: witness claude_code's prompt_submit and stop at 2.1.263 #114 moved to 2.1.263; it now reads both versions from the data.tests/test_adapter_kimi_code.py: W58's and W56's appended tests both kept.Checks
pytest -q: 1594 passed, 4 skipped (3.11); 1591 passed, 7 skipped (3.10)ruff check .andruff format --check .passtests/check_stdlib_only.py)examples/generate.py --check: up to dategit commit -s)Notes for the reviewer
The
full_matrixCI run has been dispatched on this branch as well as on #120's; that Windows job is the one check Linux cannot stand in for, and it is the reason this set gates the next release.🤖 Generated with Claude Code
Generated by Claude Code