matrix: witness claude_code's prompt_submit and stop at 2.1.263 - #114
Merged
Merged
Conversation
Both gates were re-run against the real CLI with the fixed experiment kit
(--driver "claude -p {prompt} --permission-mode acceptEdits"), twice per gate
with identical results. Every trial reads agrees or unasserted. The stop
gate's deny now reads block: true, which is what W54's re-fire observable
made visible.
block, rewrite and fail_mode on both cells carry per-claim records at
live-run / 2.1.263 / 2026-09-07 naming the driver, and both cells gain
silence_means: allow, timeout_fail_mode: open and unknown_verb_means: allow
with records of their own. All three blocking events this row claims now
assert the same six fields from the same round, so the row's own
verified.version moves to 2.1.263 and verified.date to 2026-09-07.
The row narrative gains the two facts the run established: a top-level
{"decision": "block"} at Stop is honoured headless, refusing the agent
permission to finish and sending it round again; and Claude Code caps that
continuation loop at eight re-fires -- nine Stop hook invocations in all --
before ending the turn regardless.
data/vendors/claude_code.json and examples/generated/claude_code.md are
regenerated from the row rather than hand-edited.
One test moved with the data. test_a_recognized_field_the_cell_does_not_carry
_is_unasserted read its absent field off claude_code's stop cell, which now
carries all three optional claims. It reads post_tool instead: a cell the
experiment kit cannot gate, and so one that never grows an optional claim
from a run.
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jothimani-rajendran
marked this pull request as ready for review
September 7, 2026 23:10
This was referenced Sep 7, 2026
jothimani-rajendran
added a commit
that referenced
this pull request
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #113. With the stop gate's observable fixed, both remaining claimed events
were re-run against the real CLI and merged into
claude_code's row by hand, the way W53did
pre_tool.The runs
tools/experiment.py --driver "claude -p {prompt} --permission-mode acceptEdits" --agent-version 2.1.263, twice per gate (table run, then--reportrun), identicalresults both times. Every trial reads
agreesorunasserted;baseline_okreadsunrecordedby design, being a run-health check rather than a vendor claim.blockfail_moderewritesilence_meanstimeout_fail_modeunknown_verb_meansThe stop gate's
denynow readsblock: true("the agent was made to continue") — thedisagreement #113 diagnosed as an instrument fault, not an agent one.
Two readings are worth naming rather than burying. At
stop,transformmeasured "theoriginal input ran; the rewrite was ignored". At
prompt_submitit measured "nothing ran:the rewrite was refused or degraded to a block" — the 0/0 shape #113 stopped scoring as
null. Both meanrewrite: false, which is what the cells already asserted, but theyreach it by different routes.
The merge
block,rewrite,fail_modeon both cells: per-claim records atlive-run/ 2.1.263/ 2026-09-07, each naming the driver and gate on one line.
silence_means: allow,timeout_fail_mode: open,unknown_verb_means: allow, each with its own record.verified.version→ 2.1.263,verified.date→ 2026-09-07, now that all threeclaimed events are witnessed at one build.
basisstayslive-run;observedalreadylisted all three.
Claude Code caps the continuation loop at eight re-fires (nine Stop hook invocations in
all, reproduced across two runs).
to_evidence()was not used. The edit is a targeted key update, dumped with the file'sown
indent=2+ensure_asciisettings — verified byte-stable on an unmodified roundtrip before editing, so the diff is only what changed.
Derived artifacts
data/vendors/claude_code.json(viatools/recount_vendor_config.py --write) andexamples/generated/claude_code.md(viaexamples/generate.py) are regenerated from therow, not hand-edited.
test_entries_match_recountandtest_committed_pages_match_what_the _library_producesboth require it.One test moved with the data
test_a_recognized_field_the_cell_does_not_carry_is_unassertedread its absent field offclaude_code'sstopcell, which now carries all three optional claims. It readspost_toolinstead — a cell the experiment kit cannot gate, and so one that never grows anoptional claim from a run. Still a real trial result, re-pointed at a different cell.
watch_versions: three fresh, nine unchanged
The three claimed events read fresh, which is what this PR set out to do. The nine
observation-only events still read
driftedand should: they were not re-run, theirper-claim records honestly say 2.1.247 / 2026-08-28, and making them read fresh would mean
either stamping 2.1.263 on cells nobody measured or deleting the records that pin them.
Either would turn a row-level version bump into evidence for a claim with none — the exact
failure this row's per-claim evidence exists to prevent. Left as they are.
Tests
Full suite green. Seven failures on this machine (kimi adapter, examples, git-hook exec
bit, install paths) reproduce on a clean
main, are Windows-specific, and are the subjectof the companion PR.
ruff check .,ruff format --check .andtests/check_stdlib_only.pyclean.🤖 Generated with Claude Code