Merge pull request #145 from open-coder-ai/claude/armed-probe-verb

Promote the experiment probe into the package behind `agentseam probe`

Author: jothimani-rajendran

.github/workflows/probe.yml

@@ -0,0 +1,63 @@
+name: Probe (recorded)
+
+# Replays every committed witness recording through the real probe classifier and fails the
+# job if any trial disagrees with what src/agentseam/data/matrix.json claims. No agent CLI,
+# no network, no credentials -- the recorded driver only -- so this is the one probe check
+# every push and PR can run for free (armed initiative, wave 1, T4).
+#
+# This does not re-witness anything -- that needs a real, installed agent (see
+# docs/coverage-gaps.md and tools/watch_versions.py, which notices when a re-witness is due).
+# It answers a narrower question: does the matrix still agree with what was already
+# witnessed, on this change, right now? A `DISAGREES` row means a change made a recorded
+# claim untrue, and this job fails loudly rather than silently passing (contract invariant 4:
+# silence is a measurement, not a pass -- see tools/probe_ci.py's own summary line).
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    paths:
+      - "src/agentseam/probe/**"
+      - "src/agentseam/data/recordings/*.json"
+      - "src/agentseam/data/matrix.json"
+      - "src/agentseam/matrix.py"
+      - "src/agentseam/matrix_data.py"
+      - "src/agentseam/matrix_evidence.py"
+      - "src/agentseam/matrix_terms.py"
+      - "src/agentseam/evidence_report.py"
+      - "src/agentseam/recordings.py"
+      - "tools/probe_ci.py"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-python@e9d6f990972a57673cdb72ec29e19d42ba28880f # v6
+        with:
+          python-version: "3.13"
+
+      - name: Replay every recording against the matrix
+        # pipefail so a disagreement's nonzero exit fails the step even through `tee` --
+        # the whole point of this job (see the module docstring for why exit 0 is unsafe).
+        run: |
+          set -o pipefail
+          python3 tools/probe_ci.py | tee probe.txt
+
+      - name: Summarise
+        if: always()
+        run: |
+          {
+            echo '## Probe (recorded driver)'
+            echo
+            echo '```'
+            cat probe.txt
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"

CONTRIBUTING.md

@@ -96,6 +96,9 @@ python3 tools/experiment.py run --agent <agent> --report \
     --agent-version <version> --reporter @yourhandle > report.json
 ```
 
+(`agentseam probe run` from a `pip install` — same code, promoted so it ships in the wheel;
+`agentseam probe list` shows every trial.)
+
 Open an issue with the **Evidence report** template and paste the result.
 
 A few things worth knowing before you run either:
@@ -117,6 +120,12 @@ A few things worth knowing before you run either:
   `reference` driver is the vendor's documentation made executable, not a measurement, and
   the schema refuses to let it claim `live-run`. `python3 tools/verify_report.py
   report.json` shows you what a maintainer will see.
+- **"witnessed", "tested" and "recorded" are not interchangeable.** *Witnessed* is a real
+  vendor client, watched, right now (`driver: real-agent`). *Tested* is an automated check
+  that exercised the code without one (`driver: reference`) — real and useful, never the
+  same claim. *Recorded* is a witnessed run frozen to a file and replayed deterministically
+  (`driver: recorded`) — the data is witnessed, the replay is not a new witness. See
+  [docs/coverage-gaps.md](docs/coverage-gaps.md) for the gates nobody has witnessed yet.
 
 Evidence carries the reporter's handle. Age and version drift are displayed rather than
 hidden — see `agentseam matrix --evidence`. A row that says "verified against 3.17.8, 87

README.md

@@ -200,9 +200,20 @@ python3 tools/capture.py uninstall --agent cursor
 
 The probe always allows, so it cannot interfere with real work, and payloads are reduced to
 shape before anything touches disk — keys and types survive, values do not. A different
-probe, `tools/experiment.py`, does the opposite on purpose: it denies, crashes and stalls,
-so it only ever runs in a throwaway directory a harness creates and removes — never point it
-at a config you work in. See [tools/VERIFY.md](tools/VERIFY.md).
+probe, `agentseam probe` (`tools/experiment.py` from a checkout — same code, promoted so it
+ships in the wheel), does the opposite on purpose: it denies, crashes and stalls, so it only
+ever runs in a throwaway directory a harness creates and removes — never point it at a
+config you work in. See [tools/VERIFY.md](tools/VERIFY.md).
+
+A report's own `driver` and `basis` fields say how it was obtained, from a closed
+vocabulary: **witnessed** means a real vendor client, watched, right now (`driver:
+real-agent`, `basis: live-run` or `live-run-partial`); **tested** means an automated check
+exercised the code path with no vendor client at all (`driver: reference`, `basis:
+vendor-docs`) — real, useful, and never the same claim; **recorded** means a witnessed run
+frozen to a file and replayed deterministically (`driver: recorded`) — the data is
+witnessed, the replay is not a new witness. `evidence_report.validate()` refuses a report
+that claims more than its driver earned. See [docs/coverage-gaps.md](docs/coverage-gaps.md)
+for the three gates nobody has witnessed yet.
 
 ## Contributing
 

docs/coverage-gaps.md

@@ -0,0 +1,77 @@
+# Coverage gaps only a real agent can close
+
+The probe (`agentseam probe`, promoted from `tools/experiment.py` in wave 1 of the `armed`
+initiative) covers 8 trials × 3 gateable events × 16 agents. The witnessed evidence covers a
+narrow slice of that: `claude_code@2.1.263`, `pre_tool` only, and every trial except
+`escalate` (the one `PreToolUse` reading no vendor doc settles). This closes the *tooling*
+gap, not the *evidence* gap -- **a cloud session holds no vendor credentials and cannot
+produce a witnessed row.** These three gaps are yours to close, on a machine with Claude
+Code installed and authenticated.
+
+## What "witnessed", "tested" and "recorded" mean
+
+A reader must be able to tell these apart without opening the source:
+
+| Word | What it means | How you get one |
+| :--- | :--- | :--- |
+| **witnessed** | Somebody watched the real vendor client do this, on this machine, right now. | `--driver "<your headless CLI, containing {prompt}>"` |
+| **tested** | An automated check exercised the code path -- the dialect, the classifier, the schema -- without a real vendor client. Real, useful, **not** a vendor observation. | `--driver reference` (the protocol made executable) or the `pytest` suite itself |
+| **recorded** | A previously *witnessed* run, frozen to a file and replayed deterministically. The data is witnessed; the replay is not a new witness. | `--driver recorded` (the default once a recording exists) |
+
+`agentseam probe run`'s own output never blurs these: a report's `driver` field is one of
+`real-agent`, `reference` or `recorded`, and its `basis` field is one of `live-run`,
+`live-run-partial`, `vendor-docs`, `vendor-source`, `third-party-install` or `inherited`
+(`src/agentseam/matrix_terms.py`). `evidence_report.validate()` refuses a report from the
+`reference` driver that claims a live basis -- that is the one invariant this whole
+initiative exists to enforce (contract invariant 1).
+
+## The three gaps
+
+1. **`escalate` at `pre_tool`.** The recording covers seven of eight trials at this gate.
+2. **Every trial at `prompt_submit`.** No agent has ever been witnessed at this gate.
+3. **Every trial at `stop`.** No agent has ever been witnessed at this gate, including the
+   sentinel re-fire asymmetry that gate's own docs describe
+   (`src/agentseam/probe/experiment.py`'s `_blocked()`).
+
+## Close one gap
+
+Pick a row from [`docs/witness-skeleton.json`](witness-skeleton.json) and run its command,
+replacing the placeholder driver with your own headless invocation:
+
+```bash
+agentseam probe run --agent claude_code --event pre_tool --trial escalate \
+    --driver "<your headless claude_code invocation, containing {prompt}>" \
+    --agent-version <version> --record --report --reporter @yourhandle \
+    > escalate-pre_tool-report.json
+
+agentseam probe run --agent claude_code --event prompt_submit \
+    --driver "<your headless claude_code invocation, containing {prompt}>" \
+    --agent-version <version> --record --report --reporter @yourhandle \
+    > prompt_submit-report.json
+
+agentseam probe run --agent claude_code --event stop \
+    --driver "<your headless claude_code invocation, containing {prompt}>" \
+    --agent-version <version> --record --report --reporter @yourhandle \
+    > stop-report.json
+```
+
+`--record` appends to (never replaces) `data/recordings/claude_code@<version>.json`, so
+running all three builds one recording covering every gate. `--report` prints the
+evidence-report shape alongside it -- the two are independent flags and combine safely; the
+`.json` files above are what you paste into the **Evidence report** issue template, or add
+directly to a PR (`data/recordings/*.json` plus the `data/matrix.json` per-claim `test`
+pointers the recording backs).
+
+`docs/witness-skeleton.json`'s `report` blocks are placeholders on purpose: replace the
+blank fields with your actual `--report` output, never hand-type a `basis`, `date` or
+`version`. `tests/test_coverage_skeleton.py` asserts the committed skeleton fails
+`evidence_report.validate()` exactly as it stands, and a genuinely filled-in block
+validates -- so the file can never be mistaken for evidence, only for the shape evidence
+takes.
+
+## What this is not
+
+A run against `reference` or `recorded` never counts as closing one of these gaps -- only a
+run against the real, installed `claude_code` does. See
+[CONTRIBUTING.md](../CONTRIBUTING.md#contributing-evidence-you-do-not-need-to-write-code)
+for the full walk-through and the two ways to submit what you find.

docs/witness-skeleton.json

@@ -0,0 +1,53 @@
+{
+  "_note": "Generated by the armed initiative, wave 1 (W1, 2026-09-10). Each entry in `gaps` names one coverage gap the committed witness recording (src/agentseam/data/recordings/claude_code@2.1.263.json) does not close, the exact command that closes it, and the evidence-report shape (src/agentseam/evidence_report.py) that command's --report output takes. Copy one `report` object, run its `command`, and replace the blank fields with the real --report output verbatim -- do not hand-type a basis, date or version. tests/test_coverage_skeleton.py asserts every `report` below, exactly as committed, is REJECTED by evidence_report.validate() -- an unfilled block must never be mistaken for a witnessed one.",
+  "gaps": [
+    {
+      "gate": "pre_tool",
+      "trial": "escalate",
+      "why": "claude_code@2.1.263 covers every pre_tool trial except escalate -- the one PreToolUse reading (permissionDecision: \"ask\") no vendor doc settles, so the reference driver refuses to guess it (src/agentseam/probe/reference_agent.py raises Undocumented there).",
+      "command": "agentseam probe run --agent claude_code --event pre_tool --trial escalate --driver \"<your headless claude_code invocation, containing {prompt}>\" --agent-version <version> --record --report --reporter @yourhandle > escalate-pre_tool-report.json",
+      "report": {
+        "report_version": 1,
+        "agent": "claude_code",
+        "basis": "",
+        "date": "",
+        "driver": "real-agent",
+        "event": "pre_tool",
+        "version": "",
+        "experiments": {}
+      }
+    },
+    {
+      "gate": "prompt_submit",
+      "trial": null,
+      "why": "No trial has ever been witnessed at prompt_submit for any agent -- claude_code's block/fail_mode/etc. claims for this gate rest on vendor-docs only.",
+      "command": "agentseam probe run --agent claude_code --event prompt_submit --driver \"<your headless claude_code invocation, containing {prompt}>\" --agent-version <version> --record --report --reporter @yourhandle > prompt_submit-report.json",
+      "report": {
+        "report_version": 1,
+        "agent": "claude_code",
+        "basis": "",
+        "date": "",
+        "driver": "real-agent",
+        "event": "prompt_submit",
+        "version": "",
+        "experiments": {}
+      }
+    },
+    {
+      "gate": "stop",
+      "trial": null,
+      "why": "No trial has ever been witnessed at stop for any agent -- including the sentinel re-fire asymmetry (experiment.py's own _blocked()) that has so far only been exercised through the reference driver, never a real one.",
+      "command": "agentseam probe run --agent claude_code --event stop --driver \"<your headless claude_code invocation, containing {prompt}>\" --agent-version <version> --record --report --reporter @yourhandle > stop-report.json",
+      "report": {
+        "report_version": 1,
+        "agent": "claude_code",
+        "basis": "",
+        "date": "",
+        "driver": "real-agent",
+        "event": "stop",
+        "version": "",
+        "experiments": {}
+      }
+    }
+  ]
+}

pyproject.toml

@@ -117,6 +117,21 @@ exclude = ["*.py.tmpl"]
 "src/agentseam/contract.py" = ["PLR0913"]
 "src/agentseam/install.py" = ["PLR0913", "PLR0917"]
 "src/agentseam/install_identity.py" = ["PLR0913", "PLR0917"]
+# The probe (promoted from tools/, T1 wave 1) prints the same human table and CLI output
+# `agentseam probe` (cli.py) is a thin wrapper over -- cli.py's own T201 exemption above
+# applies equally to the functions that actually do the printing (probe/cli.py is `agentseam
+# probe`'s own dispatch; experiment_cli.py is tools/experiment.py's own). `run_trial`,
+# `record` and `run_turn` are the harness's own config-knobs functions (same shape as
+# contract.py above: one or two positional args, the rest optional and keyword-only).
+# `NoRecording` and `Undocumented` are pre-existing, widely-tested public names
+# (tools/experiment_kit tests import and `pytest.raises` on them by name); renaming for
+# N818's Error suffix churns every caller of a promoted module for no behavioural gain.
+"src/agentseam/probe/cli.py" = ["T201"]
+"src/agentseam/probe/experiment.py" = ["PLR0913"]
+"src/agentseam/probe/experiment_cli.py" = ["T201"]
+"src/agentseam/probe/experiment_report.py" = ["T201"]
+"src/agentseam/probe/recorded_driver.py" = ["T201", "PLR0913", "N818"]
+"src/agentseam/probe/reference_agent.py" = ["PLR0913", "N818"]
 # TODO(lint-adoption): this wave's Sonar/Checkstyle/FindBugs baseline (coding-standards.md
 # §3) was measured and fixed for src/ only -- 100 findings at 2026-09-02, enumerated in
 # plan/spine-a/reports/w51.md. tests/, tools/, examples/ and docs/'s two asset scripts were

src/agentseam/cli.py

@@ -10,13 +10,13 @@
 import textwrap
 from datetime import date
 
-from . import __version__, adapters, recordings
+from . import __version__, adapters, probe, recordings
 from . import install as install_mod
 from . import instructions as instructions_mod
 from . import packaging as packaging_mod
 from . import permissions as permissions_mod
 from . import staleness as staleness_mod
-from .contract import EVENTS
+from .contract import EVENTS, PRE_TOOL
 from .matrix import MATRIX, enforcement_level
 
 
@@ -261,6 +261,7 @@ def _main(argv=None):
     d = sub.add_parser("doctor", help="what is wired here; flag stale capability claims")
     d.add_argument("--repo", default=".")
     d.set_defaults(fn=_cmd_doctor)
+    probe.cli.add_subparser(sub, default_event=PRE_TOOL)  # armed: probe/cli.py owns its own wiring (review budget)
 
     i = sub.add_parser("install", help="wire a handler command into an agent's config")
     i.add_argument("agent", help="agent name, or 'all'")

src/agentseam/probe/__init__.py

@@ -0,0 +1,32 @@
+"""Armed: measures what an agent's hooks actually enforce, instead of asserting it.
+
+Promoted from tools/ (wave 1, T1) so it ships in the wheel behind `agentseam probe`
+(cli.py) instead of being a developer-only script. tools/*.py at the old paths are now
+thin shims over these modules -- see tools/experiment.py's own docstring.
+"""
+
+from __future__ import annotations
+
+from . import (
+    cli,
+    experiment,
+    experiment_cli,
+    experiment_driver,
+    experiment_escalate,
+    experiment_probe,
+    experiment_report,
+    recorded_driver,
+    reference_agent,
+)
+
+__all__ = [
+    "cli",
+    "experiment",
+    "experiment_cli",
+    "experiment_driver",
+    "experiment_escalate",
+    "experiment_probe",
+    "experiment_report",
+    "recorded_driver",
+    "reference_agent",
+]

src/agentseam/probe/_shell.py

@@ -0,0 +1,29 @@
+"""The one place this package hands a command string to a shell.
+
+`agentseam-no-shell-true` bans `shell=True` across `src/`, and it is right to: the runtime
+path sits inline in a developer's agent loop, on payloads an attacker can influence. The
+probe is not that path. It is a deliberate command-runner, and the shell is the thing under
+measurement rather than an implementation detail:
+
+- `reference_agent` is Claude Code's protocol made executable, and Claude Code runs a hook by
+  handing a command string to a shell. An argv list would emulate a vendor that does not
+  exist, and every `tested`-basis row is measured against this driver.
+- The `transform` trial runs the command *as the hook rewrote it*. Measuring the rewrite means
+  running the rewrite.
+- The driver template is a command line the operator typed on their own machine.
+
+So the exception is real, and it is confined to this function rather than scattered across four
+call sites: one place to audit, and the rule keeps biting everywhere else in `src/`.
+"""
+
+from __future__ import annotations
+
+import subprocess
+
+
+def run_shell(command, *, check=False, capture_output=True, **kwargs):
+    """`subprocess.run(command, shell=True)`, defaulted to capture and never to raise."""
+    # nosemgrep: agentseam-no-shell-true -- the audited exception; see this module's docstring.
+    return subprocess.run(  # noqa: S602
+        command, shell=True, check=check, capture_output=capture_output, **kwargs
+    )