Skip to content

matrix: witness claude_code's prompt_submit and stop at 2.1.263 - #114

Merged
jothimani-rajendran merged 2 commits into
mainfrom
w55/witnessed-prompt-submit-and-stop
Sep 7, 2026
Merged

jothimani-rajendran merged 2 commits into
mainfrom
w55/witnessed-prompt-submit-and-stop

Conversation

@jothimani-rajendran

Copy link
Copy Markdown
Collaborator

Follow-on to #113. With the stop gate's observable fixed, both remaining claimed events
were re-run against the real CLI and merged into claude_code's row by hand, the way W53
did pre_tool.

The runs

tools/experiment.py --driver "claude -p {prompt} --permission-mode acceptEdits" --agent-version 2.1.263, twice per gate (table run, then --report run), identical
results both times. Every trial reads agrees or unasserted; baseline_ok reads
unrecorded by design, being a run-health check rather than a vendor claim.

trial field prompt_submit stop asserted status
deny block true true true agrees
crash fail_mode open open open agrees
transform rewrite false false false agrees
silence silence_means allow allow unasserted
timeout timeout_fail_mode open open unasserted
unknown unknown_verb_means allow allow unasserted

The stop gate's deny now reads block: true ("the agent was made to continue") — the
disagreement #113 diagnosed as an instrument fault, not an agent one.

Two readings are worth naming rather than burying. At stop, transform measured "the
original input ran; the rewrite was ignored". At prompt_submit it measured "nothing ran:
the rewrite was refused or degraded to a block" — the 0/0 shape #113 stopped scoring as
null. Both mean rewrite: false, which is what the cells already asserted, but they
reach it by different routes.

The merge

  • block, rewrite, fail_mode on both cells: per-claim records at live-run / 2.1.263
    / 2026-09-07, each naming the driver and gate on one line.
  • Both cells gain silence_means: allow, timeout_fail_mode: open,
    unknown_verb_means: allow, each with its own record.
  • Row verified.version → 2.1.263, verified.date → 2026-09-07, now that all three
    claimed events are witnessed at one build. basis stays live-run; observed already
    listed all three.
  • Row narrative extended by exactly two sentences: the headless Stop block is honoured, and
    Claude Code caps the continuation loop at eight re-fires (nine Stop hook invocations in
    all, reproduced across two runs).
  • to_evidence() was not used. The edit is a targeted key update, dumped with the file's
    own indent=2 + ensure_ascii settings — verified byte-stable on an unmodified round
    trip before editing, so the diff is only what changed.

Derived artifacts

data/vendors/claude_code.json (via tools/recount_vendor_config.py --write) and
examples/generated/claude_code.md (via examples/generate.py) are regenerated from the
row, not hand-edited. test_entries_match_recount and test_committed_pages_match_what_the _library_produces both require it.

One test moved with the data

test_a_recognized_field_the_cell_does_not_carry_is_unasserted read its absent field off
claude_code's stop cell, which now carries all three optional claims. It reads
post_tool instead — a cell the experiment kit cannot gate, and so one that never grows an
optional claim from a run. Still a real trial result, re-pointed at a different cell.

watch_versions: three fresh, nine unchanged

claude_code/pre_tool       fresh    2.1.263  2.1.263  0d
claude_code/prompt_submit  fresh    2.1.263  2.1.263  0d
claude_code/stop           fresh    2.1.263  2.1.263  0d
claude_code/post_tool      drifted  2.1.247  2.1.263  10d
... (8 more observation-only events, all drifted at 2.1.247)

The three claimed events read fresh, which is what this PR set out to do. The nine
observation-only events still read drifted and should: they were not re-run, their
per-claim records honestly say 2.1.247 / 2026-08-28, and making them read fresh would mean
either stamping 2.1.263 on cells nobody measured or deleting the records that pin them.
Either would turn a row-level version bump into evidence for a claim with none — the exact
failure this row's per-claim evidence exists to prevent. Left as they are.

Tests

Full suite green. Seven failures on this machine (kimi adapter, examples, git-hook exec
bit, install paths) reproduce on a clean main, are Windows-specific, and are the subject
of the companion PR. ruff check ., ruff format --check . and
tests/check_stdlib_only.py clean.

🤖 Generated with Claude Code

Both gates were re-run against the real CLI with the fixed experiment kit
(--driver "claude -p {prompt} --permission-mode acceptEdits"), twice per gate
with identical results. Every trial reads agrees or unasserted. The stop
gate's deny now reads block: true, which is what W54's re-fire observable
made visible.

block, rewrite and fail_mode on both cells carry per-claim records at
live-run / 2.1.263 / 2026-09-07 naming the driver, and both cells gain
silence_means: allow, timeout_fail_mode: open and unknown_verb_means: allow
with records of their own. All three blocking events this row claims now
assert the same six fields from the same round, so the row's own
verified.version moves to 2.1.263 and verified.date to 2026-09-07.

The row narrative gains the two facts the run established: a top-level
{"decision": "block"} at Stop is honoured headless, refusing the agent
permission to finish and sending it round again; and Claude Code caps that
continuation loop at eight re-fires -- nine Stop hook invocations in all --
before ending the turn regardless.

data/vendors/claude_code.json and examples/generated/claude_code.md are
regenerated from the row rather than hand-edited.

One test moved with the data. test_a_recognized_field_the_cell_does_not_carry
_is_unasserted read its absent field off claude_code's stop cell, which now
carries all three optional claims. It reads post_tool instead: a cell the
experiment kit cannot gate, and so one that never grows an optional claim
from a run.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jothimani-rajendran
jothimani-rajendran marked this pull request as ready for review September 7, 2026 23:10
@jothimani-rajendran
jothimani-rajendran merged commit 716bf60 into main Sep 7, 2026
14 checks passed
jothimani-rajendran added a commit that referenced this pull request Sep 8, 2026
…en-waves

integration: the seven reviewed waves of 2026-09-07 (#114 #115 #116 #118 #119 #120 #117)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants