Repository navigation
Step 1 emits its prompt instead of executing it, and NFR-1 gets a measurement driver - #295
Merged
Merged
Conversation
…de needed MeasurementSession has always had two modes. Driven mode invokes a synthetic callable and exists only for self-test. Observer mode -- the one a real NFR-1 run needs -- requires an external driver to call open(), mark_tool_call() once per real tool call, then close(). No such driver shipped, so the harness could report INDETERMINATE and nothing else, whatever else was fixed. That is why #259 stayed open with the harness already built. The obvious place to call mark_tool_call() was the PostToolUse hook, which fired once per tool call -- and that hook was deleted by the very release this measurement exists to justify. The signal used instead is written by Claude Code itself and survives that deletion: the session transcript, whose assistant records carry message.content[] blocks of type "tool_use". tests/nfr1/driver.py - TranscriptTail reads only what was appended after construction, in binary so offsets are true byte offsets. Text-mode tell() returns an opaque cookie that was being compared to st_size as if it were one. Partial lines are buffered as bytes and decoded only once whole, so a record split across two flushes -- or split mid-character -- is parsed once and completely. - Append-only is verified rather than assumed: the bytes before the offset are re-read each poll and compared. A mismatch raises TranscriptRewritten and the window is void. Re-scanning from the start would sweep up records written before the window opened with no way to tell them apart, so a void observation stays void rather than becoming a number nobody can defend. - drive() applies no default wall-clock deadline. Bounding a measurement by time is the temporal proxy NFR-2 exists to remove, so both bounds are opt-in. tests/nfr1/cli.py - --observe --phase cold|warm runs a single phase in observer mode against the live transcript. One phase per invocation is deliberate: a cold count is only cold if the window opens before the session has issued anything. - A rewritten transcript is reported as void_reason with no count, never as a short window. Verification, not review: - 40 tests. Every positive assertion is paired with a negative, and every filter with a specificity test proving it does not swallow what it must keep. - 12-mutation run: 12 caught, 0 survivors, no hangs, file restored byte-exact. - Run against this live session: 10/10 real tool calls observed, with real toolu_ ids and timestamps; the window closed on the count, not on a bound. Four defects that review did not find are recorded in docs/REVIEW-INDEX.md as corrections 55-57 plus the resolver bug: a rewrite check that only caught shrinkage (found by a paired assertion), two assertions that could not fail (found by mutation), a test that hung instead of failing when its behaviour was mutated away (found by a bounded harness), and a transcript path resolver that computed a directory which does not exist (found by comparing it to the real filesystem). This does not close #259. It makes the harness capable of a measurement it could not previously take. The first real observation now returns INDETERMINATE for a new reason -- 0 plugin-attributable processes, but 88 the registry cannot name, 22 of them Claude Code's own statusline -- which is a scope decision, not a fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ch at all
Two separate ways NFR-1 could not fail, one of them older than this sprint.
1. PLUGIN MARKERS COULD NOT MATCH A WINDOWS PROCESS AT ALL
build_default_registry() normalises plugin_root-derived markers to forward
slashes and lowercase. ComponentSpec.matches() compared them against
record.search_text() unnormalised, and Windows reports command lines and exe
paths with backslashes. A marker of "c:/users/.../plugins/cwe" is not a
substring of "C:\Users\...\plugins\cwe\entry.py".
So plugin_count was structurally zero on Windows regardless of what the plugin
spawned. Any prior "0 processes attributable to the plugin" result measured
nothing. The harness's own --self-test never caught it because its synthetic
marker carries no path separator and so never exercised the asymmetry.
Both sides are now normalised: markers at construction, process text in
matches(). Normalising only one side is how a marker dies silently -- it stops
matching, the component collects nothing, and that reads exactly like "this
component spawned no processes".
2. A BROAD MARKER ON AN UNCAPPED ROLE COULD SWALLOW A PLUGIN SPAWN
attribute() direct-matched the process before walking its ancestry, and took
the first matching component in registration order. A plugin-spawned shell
whose own command line does not name the plugin would direct-match any
component declaring a broad marker and never reach the ancestry walk that
would have charged it to the plugin.
That is the damage MAX_PERMITTED_EXCLUSIONS = 1 exists to prevent, reachable
through ROLE_OBSERVED, which has no cap and needs no justification.
attribute() now resolves ROLE_PLUGIN_COUNTED first against the process AND its
full ancestor chain, before any other component's direct match. Shared
_match_role/_walk_ancestry primitives keep the two searches from drifting.
BASIS_DIRECT/BASIS_ANCESTRY semantics, via_pid, the depth guard and the
AttributionResult JSON shape are unchanged.
3. TWO NARROW OBSERVED COMPONENTS
statusline_hook (marker "statusline-command.sh") and claude_code_host (marker
"claude.exe"). Console hosts and tool-call shells reach claude_code_host by
ANCESTRY only; they carry no marker of their own, because bare interpreter
names are exactly the shape point 2 exists to stop.
node.exe and chrome.exe are deliberately left unattributed. Measurement showed
the CLI here runs as claude.exe and the node.exe processes were unrelated dev
servers with no Claude Code ancestor, so marking them would have been wrong on
this machine rather than merely unspecific. An honest INDETERMINATE beats a
guessed marker.
INTERLOCK, stated so nobody removes half of it: the plugin's entry points are
themselves spawned by claude.exe, so every plugin process has the
claude_code_host marker in its ancestry. Plugin-first precedence is the only
thing stopping every plugin process being charged to that OBSERVED component.
Declaring claude_code_host without the precedence fix would be strictly worse
than declaring nothing.
VERIFICATION
- 2523 passed / 50 skipped / 11 xfailed / 0 failed, pytest exit 0, zero
FAILED or ERROR lines.
- 3/3 mutations caught, zero survivors, file restored byte-for-byte. The
plugin-first DIRECT match initially SURVIVED its own tests, because
KEY_PLUGIN is registered first and first-match-wins found it anyway; the
property was unproven until an adversarial registry that registers an
OBSERVED component before the plugin was added.
- cli.py --self-test still returns FAIL with plugin_count=1. The metric
remains failable, which is the whole point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
58: plugin markers were normalised to forward slashes while the process text they were compared against kept Windows backslashes, so the plugin marker was a substring of nothing and plugin_count was structurally zero. Older than this sprint, and invisible to --self-test because its synthetic marker has no path separator. Any prior Windows NFR-1 zero measured nothing. 59: plugin-first precedence landed green while still coupled to registration order -- deleting the plugin-first direct match broke no test, because KEY_PLUGIN happens to be registered first. The property was unproven by its own suite until an adversarial registry was added. 60: fixing one side of the normalisation killed any backslash marker silently. Both sides now normalise. 61: the orchestrator's brief claimed node.exe was the Claude Code host. It is claude.exe here; the node processes were unrelated dev servers. The implementing agent measured it and refused the brief, which is what briefs should invite. Also records the interlock: plugin entry points are spawned BY claude.exe, so plugin-first precedence is the only thing keeping every plugin process from being charged to the claude_code_host OBSERVED component. The two must not be separated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NFR-1 could not reach a verdict on any machine doing real work. Every process that
matched no component forced INDETERMINATE, and unrelated software -- a browser,
somebody's dev server -- can never match a component, so the condition held
permanently no matter how good the measurement was.
But those two situations carry opposite weight. A process whose ancestry was
walked back to something that already existed before the window opened, with
every hop present and readable and no plugin marker anywhere on the chain, has
been SHOWN not to descend from the plugin. Only a walk that could not finish
leaves the question open.
WHAT TERMINATES A PROOF, AND WHY IT IS NOT "A ROOT"
The baseline is the terminator, not a root, and the distinction is load-bearing.
ppid is documented as None when the backend WITHHELD it, and a chain almost
always ends at a parent that exited and is in no snapshot. Neither is a root, and
treating either as one manufactures proof out of missing data. Reaching a process
that predates the window is different: NFR-1 counts what the plugin spawned
DURING the window, so a pre-existing non-plugin ancestor settles everything below
it.
Proof requires a baseline to be supplied, so callers that pass none keep the old
conservative behaviour and nothing silently becomes provable.
FOUR WAYS THE PROOF COULD HAVE BEEN WRONG, ALL CLOSED
- The process's own identity. The proof is about ANCESTRY and says nothing
about the process itself, so a process observed with no command line and no
executable path might BE the plugin. This is not hypothetical: the shipped
--self-test, which spawns a marked child and requires FAIL, returned PASS
once with the marked child neither charged nor reported unknown. The proof
now requires the process's own identity to be readable. An image name is not
enough -- a plugin entry point runs as python.exe like everything else.
- Pid reuse. The index entry for a parent pid may be a LATER process that
inherited the number, sending the walk up a chain that never existed. A
parent cannot be created after its child, so that ordering is checked.
- An unreadable hop. It clears the provable flag but no longer STOPS the walk:
a plugin marker may sit above it, and missing that would under-report the one
thing NFR-1 counts. Erring toward finding the plugin is the safe direction.
- Ordering. The baseline is tested after an ancestor's markers, so a plugin
sitting ON the baseline is charged rather than proved.
ANCESTRY INDEX NOW INCLUDES WHAT THE SAMPLER SAW
It was built from the endpoint snapshots only, while the records being attributed
come from the sampler, which by design sees processes in neither snapshot. When a
short-lived shell's parent was also short-lived, the parent was absent and the
chain broke at the first hop. That is why a live window reported 63 processes as
unknown while all 63 carried a usable parent pid.
VERIFICATION
- 2548 passed / 50 skipped / 11 xfailed / 0 failed, pytest exit 0, zero FAILED
or ERROR lines.
- 11/11 mutations caught, zero survivors, file restored byte-for-byte. Each
mutation weakens the proof in a different direction.
- --self-test: 24 consecutive runs report FAIL, 10 idle and 14 under real load
from a concurrent full-suite run. The original PASS was NOT reproduced, so
this is recorded as closing the class the failure's shape matched, not as a
confirmed repair of that specific race.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…caught before it shipped 62: the orchestrator's own change made --self-test report PASS where it must report FAIL. The proof is about ancestry and was being granted to a process whose own identity had not been read; such a process might BE the plugin. Not reproduced afterwards (24 runs, idle and loaded, all FAIL), so recorded as closing the class the failure's shape matched. 63: ppid None means the backend withheld the parent, not that the process is a root. Treating it as a root manufactures proof from missing data, and forced the better definition: proof terminates at a process that predates the window. 64: stopping the walk at an unreadable ancestor under-reports the plugin, since a marker may sit above it. Caught by a test written in the previous round. 65: the ancestry index was built from endpoints while the attributed records came from the sampler, so short-lived parents were missing and their children's chains broke at the first hop. Also records the third live measurement: 252 of 345 attributed, 12 proved, plugin-attributable 0 in all three deltas, 81 still unknown. States plainly that the runs are not comparable and that 65 of the 81 are transient intermediates, which is a limit of sampling rather than a defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t can be valid
Every ordering available before this was either uncountable or invalid.
The criterion requires ten tool calls inside ONE response turn. The retained Stop
hook (ADR-010) fires at every turn boundary, so launching the driver in one turn
and issuing the calls in the next produces a window the turn-boundary guard
correctly rejects -- which is exactly what happened on the first real attempt:
stop-notifier.log grew from 2,239,998 to 2,240,296 bytes mid-window and the
measurement was INVALID.
The alternative was no better. Issuing the calls in the same assistant message as
the launch puts them in a record already written before the tail exists, and a
tail anchored at end-of-file cannot see them.
--from-current-record anchors the tail at the start of the transcript's last
assistant record instead, so one message can carry both the launch and the calls.
The file is searched backwards in growing windows rather than read whole, since a
real transcript runs to tens of megabytes.
--skip-leading N discards the first N observed calls. The launching call began
before the window opened, so counting it would credit the window with a call it
did not contain. The tail cannot know its own tool-use id, so the exclusion is
explicit rather than inferred.
RESULT: THE FIRST VALID MEASUREMENT THIS HARNESS HAS PRODUCED
tool calls 10 / 10
turn boundary clean -- all three witnesses silent
plugin 0 in all three deltas
verdict INDETERMINATE on 56 unknowns (49 broken chains, 7 the OS
would not describe)
Still not a PASS, and it cannot be one yet: structural gates report
NOT_MEASURABLE because no plugin is installed, so plugin=0 remains trivially
true. What changed is that the measurement is now VALID rather than rejected.
VERIFICATION
- 8/8 mutations caught, zero survivors, file restored byte-for-byte.
- One mutation initially SURVIVED: the widening-window test placed its target
at offset 0, where "found it" and "gave up" are the same answer. Fixture
corrected so the expected offset is non-zero, and the test now asserts that.
- A fixture used Path.write_text, which translates newlines on Windows and
shifted every asserted offset. Real transcripts are LF-only, measured against
a live one; fixtures now write bytes, and a CRLF case is pinned separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wn token policies 66: a mutation survived because the fixture's correct answer (byte offset 0) was identical to the function's failure value. A test whose expected result equals what a broken implementation returns cannot detect that break. 67: a fixture used Path.write_text, which translates newlines on Windows and shifted every offset it asserted on. The finder was right, the test was wrong. Real transcripts measured as LF-only; fixtures now write bytes. Sixth instance of silent newline translation in this record, never once the code's fault. 68: the orchestrator was not following docs/policies/file-management-policy.md while writing this very file -- four full-file read+write appends to a 700-line document, one Write where Edit belonged, and re-reads past the cache_after_n_reads threshold. The full-file rewrites happened inside a subprocess so the conversation paid no tokens, which is why it went unnoticed, and a full-file rewrite is also the mechanism behind every newline flip recorded here. Entry written with the policy's own pattern: tail the last lines, then Edit to append. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s basename build_default_registry appended os.path.basename(plugin_root) as a marker. This plugin's directory is literally named 'plugin', so the marker was the single word 'plugin' -- matching any command line that mentions it. 'chrome.exe --disable-plugins' would have been charged to the plugin. That direction produces false FAILs rather than false passes, so it never corrupted a measurement and nothing matched it during the first real run. But a metric that can fail for the wrong reason is no better than one that cannot fail at all, and this was shipped. The basename existed for a reason: the absolute-path marker only matches the plugin at exactly this location, and a shorter relocatable marker is wanted too. Replaced with the last TWO path components -- 'claude-workflow-engine/plugin' -- which stays relocatable while being distinctive. A root with fewer than two components yields no such marker, because there is nothing to qualify it with. Markers with the real installed root are now: claude_plugin_root claude-workflow-engine-plugin c:/users/.../claude-workflow-engine/plugin claude-workflow-engine/plugin Tests are behavioural rather than syntactic: an unrelated browser flag mentioning plugins is not charged, the real plugin path still is, a relocated checkout still is, and no derived marker is a bare generic word. Found while checking whether plugin_count=0 from the first plugin-grounded measurement was meaningful. It was -- and this makes it mean the right thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…one report
A genuinely cold phase can only be observed at the very start of a fresh session,
so cold and warm are necessarily two separate invocations. build_report takes
Measurement objects and nothing could read one back from JSON, so two observed
phases could not be reported together at all -- a gap recorded earlier rather
than worked around.
WHAT IS RESTORED, AND WHAT DELIBERATELY IS NOT
AttributionResult.to_dict() serialises counts plus the unattributed processes in
full, not every attribution. The live object therefore cannot be reconstructed,
and a class claiming to do so would be inventing the difference.
RestoredMeasurement keeps the recorded payload verbatim instead and answers
verdict() from the numbers that were actually written, flagging itself as restored
in to_dict().
THE DRIFT THIS AVOIDS
The verdict ladder moved out of Measurement.verdict() into compute_verdict(),
which both the live and the restored path now call. A second copy would have been
free to disagree, and the direction it would drift is toward PASS, because that
branch has the fewest conditions.
MISSING EVIDENCE RAISES
Any absent key -- phase, tool call count, turn boundary, or any of the three union
counts -- raises rather than defaulting. A count defaulted to zero turns absent
evidence into a pass, which is the one outcome this harness exists to prevent.
Supplying the phases the wrong way round raises too, since that would silently
label a warm count as cold.
cli.py --report --cold-json A --warm-json B assembles both into one NFR1Report.
No combined figure is exposed: cold-start benchmarking is a named anti-pattern and
a blend of the two describes neither.
VERIFIED
- 182 NFR-1 tests pass.
- Round-tripped a real recorded measurement: verdict and reasons both identical.
- Paired negatives throughout: every missing key raises, a complete payload does
not, and a report missing a phase refuses a verdict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…unded 69: the plugin marker set included the bare word 'plugin', derived from the root's basename. Errs toward false FAILs, so no measurement was corrupted, but a metric that can fail for the wrong reason is no better than one that cannot fail. 70: the orchestrator declared the plugin uninstalled, then not loaded, and was wrong both times. installed_plugins.json and 'claude plugin list' both stayed empty while the plugin was demonstrably running -- neither is the source of truth for a directory-sourced marketplace. The skills appearing mid-turn settled it. 71: the measurement's own ten tool calls are a dominant source of the processes it cannot attribute -- one new process in 25 idle seconds against eighty across the window. With the statusline also parked, observed dropped 144 to 81. Neither is a harness defect; they bound what sampling can establish on Windows. Also records exactly what the first plugin-grounded measurement established, why its zero is no longer trivially true, and that both live-settings changes were authorised and reversed -- settings.json verified byte-for-byte identical afterwards, and the Stop hook deliberately left alone because the turn-boundary guard depends on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…en by pasting The full invocation is long enough that pasting it into a terminal wraps the line, which silently drops an argument -- observed live as 'argument --plugin-root: expected one argument' followed by the path being executed as its own command. scripts/bin/nfr1-observe.cmd <cold|warm> <out.json> resolves the repo from its own location and supplies --from-current-record, --skip-leading 1 and --plugin-root, so the operator pastes one short line instead of four hundred characters. Its header states the constraint that the flags cannot: this must be launched BY a Claude Code session as a background tool call, not typed into a shell on its own. The observer counts the tool calls that session issues, so with no session driving it there is nothing to count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… one A real cold run counted zero tool calls and polled its entire 267-second budget. The transcript it tailed was the right one, its session id matched, and it held sixteen tool_use blocks. It still saw none of them. WHY --from-current-record was built on a wrong assumption about the transcript format. Claude Code writes ONE assistant record per tool call, and writes all of a response's records BEFORE the tools execute. Eleven calls issued in a single response are eleven records, and measured timestamps show every one of them was on disk before the launcher's own process opened the window -- the launcher's record 20.0s before, echo c10 16.5s before. So "anchor at the last assistant record" landed on `echo c10`, past every call in the batch. --skip-leading 1 then consumed that one too, and the driver waited for calls that had already been written. THE FIX find_launching_record_offset() searches backwards for the last assistant record containing a distinctive substring of this invocation -- the --json-out path, which is unique per run -- and anchors there. Taking the LAST match means a rerun that reuses an output path anchors on its own launch rather than an earlier one. When the needle is absent the CLI falls back to the previous behaviour and says so: anchored_at now reports launching_record, last_assistant_record_fallback, or end_of_file, rather than asserting one shape. VERIFIED AGAINST THE TRANSCRIPT THAT FAILED Reconstructed to its state at observer start (84,842 bytes, through echo c10): old anchor (last assistant record) = 83376 -> 0 tool calls counted new anchor (launching record) = 68630 -> 10 tool calls counted Tests assert both halves, so the regression is pinned rather than described. Paired negatives: an absent needle returns None so the caller can fall back, an empty needle matches nothing rather than everything, and a non-assistant record carrying the needle is not chosen. KNOWN LIMIT, stated because it is reachable: if a later call in the SAME response also mentions the output path, the anchor moves past the batch again. The measurement prompt must not ask for the file to be read in the same response that launches the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cold run that finally counted 10/10 measured a window **0.19 seconds** long, with a single sampler poll and every process delta at zero. It bracketed none of the execution it existed to bracket. WHY A tool_use block records an intention and is written BEFORE the tool runs. A tool_result block, carried on a user record, records the completion. Anchored at its launching record, the driver saw all ten tool_use blocks the instant it opened -- they were already on disk -- counted them, and closed. Measured on that transcript: the ten uses were written between +0.0s and +7.9s, and their results arrived between +13.8s and +19.7s. Everything the window was meant to observe happened after it had already shut. THE FIX The tail now tracks which calls were issued after the anchor and reports one event per matching tool_result. The window therefore closes when the tenth call has actually finished, spanning the execution instead of the intent. A result for a call issued before the anchor is ignored, so only this window's own calls can complete it, and each completion is reported once. --skip-leading still drops the launching call first, so the launcher's own result -- which cannot arrive while it is still running -- can never complete the window. VERIFIED AGAINST THE TRANSCRIPT THAT FAILED Reconstructed to the state at observer start: counting intentions -> 10 events immediately (the 0.19s window) counting completions -> 0 events, then 10 once the results land Both halves are pinned by tests, including the paired negative that reproduces the old immediate return. The CLI wiring fixtures were writing tool_use records only, so they had stopped exercising what the driver does; they now record results too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…calls 225 seconds was too short once the probes had to be long-running. They must outlive the observer's own startup -- Bash spawn, Python boot, psutil import and a 450-process snapshot, about 20 seconds measured -- because a driver that is itself one of the calls it counts opens its window after the fast ones have already finished. That is what produced a 0.2-second window measuring nothing. If the harness runs the probes sequentially rather than in parallel their total reaches several minutes, and a budget expiring first would report a short count as though the calls had never happened. 3600 polls at 0.25s gives 15 minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…et one Overall verdict INDETERMINATE, and that word is doing precise work: plugin attributable count is 0 in every delta of both phases, both structural gates PASS, both windows stayed inside a single response turn, and 10/10 tool calls were counted in each. What blocks a verdict is ten processes -- nine of them Windows system processes whose command lines an unelevated observer cannot read. A PASS is therefore one elevated run away, which is worth attempting before any amendment to the criterion, because if it succeeds no amendment is needed. Three of the issue's four closure conditions are now met. The fourth -- that a pass has been produced -- is not, and the report says so rather than reading the zero as a result it is not yet entitled to be. The report also records the eight instrument defects that had to be fixed before the number could be trusted, every one found by attempting a real measurement rather than by review, and one finding outside NFR-1 entirely: the operator's ~/.bashrc invoked the Angular CLI on every shell startup, costing 2929ms and six processes per shell, on every tool call of every session. Now cached: 375ms. nfr1-cold5.json is kept alongside the finals because it is the run that located that shell-profile chain: 79 unknowns, 53 of them npm and ng completion processes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
harness.py carried its own closure condition: "This issue cannot close on a harness that has never produced a pass." Three of its four conditions are now met -- V2-015, V2-027, and cold and warm both measured for the first time. The fourth is not: both phases are INDETERMINATE on ten processes, nine of them Windows system processes whose command lines an unelevated observer cannot read. The owner declined the elevated run that might have cleared them, on the grounds that a criterion satisfiable only under Administrator rights is one most engineers in a normal or corporate environment could never verify. The ruling stands and is recorded rather than inferred. closure_note now carries BOTH halves -- the original bar and the amendment -- and two tests pin it, so a later rewrite cannot quietly turn "the bar was waived" into "the bar was cleared". One asserts the original wording survives alongside the amendment and that the note still says a PASS was not produced; the other asserts it names where its evidence lives, because a ruling with no reachable evidence is an assertion. The report gains a section stating exactly what the closure claims -- the plugin contributes zero processes to a session that installed and did not invoke it, measured twice under conditions the harness certifies as valid -- and exactly what it does not: a PASS, that every process was identified, or that the criterion as written is satisfiable unelevated on Windows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t to build Written from three independent surveys done against live code rather than documentation: the MCP server surface, policy conformance, and library routing. THE DIAGNOSIS Almost everything needed already exists. Very little of it is connected. Two failure shapes recur in every layer. Built, tested, never called: select_agents() is a KG scoring engine with zero production callers; SkillAgentLoader reads the exact directories a just-in-time loader would target and has zero production callers; 19 of 21 sibling MCP servers have no Python call site, including mcp-jira-api's 52 tools and mcp-figma's 47. Imports of things that do not exist, failing silently: github_mcp_server was extracted, jira_mcp_server and figma_mcp_server never existed, 7 of 9 Stop-hook spawn targets are absent repo-wide, MCPPluginLoader is absent. Every one is caught, logged and continued past -- which is why nobody noticed how many there are. WHAT IS LOAD-BEARING Two of 26 registered servers. session-mgr, and not over MCP -- the engine imports an in-engine copy directly. And push-gate. The other 24 registrations predate v2.0.0 and sit outside anything the plugin manages; the plugin's own registry holds exactly two entries. WHAT THE SURVEYS CORRECTED docs/policies/ does not mirror ~/.claude/policies/ -- 18 files have no counterpart and 6 exist only in the runtime copy, which is the one standards/selector.py reads. ADR-010, ADR-017, ADR-019 and ADR-020 have no standalone files. CLAUDE.md's MCP table sums to 137 tools while claiming 295. mcp-base, checked, has no drift. THE CONSTRAINT THAT SHAPES THE PLUGIN-OF-PLUGINS PLAN 22 MCP servers as 22 plugins each bundling .mcp.json would pass NFR-1 individually while reproducing in aggregate exactly what ADR-019 forbade. The pattern that avoids it is already built and proven by push-gate: a plugin declares a server, only an explicit user action starts one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…N correction Three owner rulings and one measured finding, folded into the integration architecture rather than appended to it. RULING: THE FLOW RUNS IN THE ACTIVE SESSION, NOT VIA claude -p The largest change here, because the engine does the opposite today: prompt_gen_expert_caller.py, todo_decomposer.py and orchestrator_agent_caller.py each spawn the claude CLI and parse stdout, two calls in Step 1 alone plus one per TODO. The shift is from "Python orchestrates LLM subprocesses" to "the live session orchestrates itself, calling Python for the deterministic parts". It also dissolves the content-injection gap rather than solving it. Under the subprocess model, injecting a skill meant serialising its body into a pipe, which is why the current code passes only names. In the active session a materialised skill in ~/.claude/skills is loaded the way any skill is -- and today's measurement showed that happens without a restart. The missing piece was never injection; it was the subprocess in the middle. RULING: EPHEMERAL CAPABILITY LIFECYCLE Materialised capabilities need to be namespaced, inventoried, reversible and self-healing after a crash. Teardown belongs in the command that materialised them, with the Stop hook only reconciling the manifest -- putting a correctness-critical cleanup inside the least reliable component in the system would be a poor trade. RULING: HARD-DISABLE THE STOP HOOK'S AUTO-PR PATH Removal, not repair, plus a test that fails if it returns. The capability already exists explicitly as /claude-workflow-engine:review. MEASURED: 159 ORPHAN .pyc FILES Compiled modules with no .py source -- 48 under tests/, 33 under langgraph_engine/, 24 under scripts/, 16 under src/mcp/. They make the codebase lie about what exists; this survey was misled by them twice. TOON Removed from live code in v1.15.x -- the three remaining .py mentions are removal notes, not calls. But docs/policies/error-recovery-policy.md still prescribed TOON compression as the recovery for context window overflow, a mechanism deleted a release earlier. Corrected to state there is NO wired recovery, with the reasoning recorded: WorkflowContextOptimizer exists and imports cleanly but its only caller is helper_nodes/, which nothing imports. Substituting one unwired mechanism for a deleted one would repeat the original mistake with a fresher name. intelligent-decision-engine-policy.md is the only policy still naming removed providers; no standard or rule does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 built its prompt from an in-repo template that had drifted badly from
the library's ORCHESTRATION_TEMPLATE.md: 198 lines against 1,996, with nine
steps lost along the way -- STEP 0 (lean KG load), STEP 0.05 (subagent dispatch
contract), STEP 0.1 (decision-tree traversal), STEP 0.5 (user consultation
gate), STEP 4.5 (thinking-level routing), STEP 6 (context engineering), STEP 7
(anti-hallucination layer), STEP 10.5 (team alignment) and STEP 10.75 (active
knowledge graph). STEP 7 is the one the policy set marks "MANDATORY for ALL
projects -- NO exceptions"; the fork could not run it at all.
_load_template() now resolves ORCHESTRATION_TEMPLATE.md through the existing
3-tier ResourceResolver chain. No resolver change was needed -- fetch_kg_file()
already resolves any path under the library root, with traversal protection and
a typed terminal failure; it was simply never called for this.
There is deliberately no fallback to the in-repo fork. A missing library now
raises LibrarySetupError naming the expected path and the override env var,
rather than silently serving a template that is missing its mandatory steps.
The eight runtime values are prepended as an authoritative grounding header
instead of being substituted in. The master is a standalone instruction
document, not a parameterised prompt: its braces are path patterns
(knowledge-graph/{slug}/agents.json) and authoring directives (Detected {N}
domains) that the orchestrator fills itself. str.replace() would corrupt them.
test_master_template_body_is_passed_through_unmodified pins the passthrough so
a later revert to substitution fails a test rather than quietly mangling the
template.
Not yet working end to end. _call_claude_cli still spawns a subprocess and
captures stdout, while the master mandates a save-and-stop protocol and forbids
printing the generated prompt. The end-to-end path was working against a stale
template and is now mid-migration against the correct one. Removing the
subprocess call and deleting the dead fork are the next two steps.
Committed from Git Bash rather than mcp-git-ops: the running MCP server still
spawns hooks as bare "bash.exe", which Windows resolves to the System32 WSL
stub before PATH. hook_shell_fix.py addresses this but needs a server restart.
The pre-commit hook ran and passed; it was not skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e451ae8 added docs/architecture/integration-architecture-2026-08-06.md and docs/reports/nfr1-measurement-2026-08-06.md without updating the counts in docs/README.md, leaving test_docs_index_matches_the_tree red on this branch. architecture 18 -> 19, reports 25 -> 26. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oaded The previous commit removed the fallback inside _load_template(), but the real fallback was one layer up and survived: on any ERROR from prompt_gen_expert_caller, task_orchestration.py set orchestration_prompt to the raw user message and carried on. A missing claude-global-library therefore ran the whole pipeline with no KG routing, no decision-tree traversal and no STEP 7 anti-hallucination layer, while every downstream step still reported success. The only trace was a single log line. Removing the inner fallback and leaving this one in place bought nothing. The two failures are not equivalent and are no longer treated as one. A failed template load is unrecoverable -- without the template there is no orchestration to perform -- and now emits error_kind=TEMPLATE_LOAD_FAILED, which the caller turns into OrchestrationTemplateUnavailable. step1_task_analysis_node is not wrapped in node_error_handler, so that propagates and stops the graph rather than being folded into a fallback result. A failed LLM call is different: the assembled prompt is still valid, so that path keeps its existing fallback and is pinned by a test, ensuring a later "any ERROR is fatal" simplification cannot make an ordinary CLI hiccup unrecoverable. Also restores three assertions in test_kg_router_exception_is_fail_open that a mis-scoped insert had orphaned into the new test class. The pipeline has no mid-flight abort gate: final_status is computed only in the summary node and nothing routes on it, so raising from an undecorated node is the loudest stop available without inventing new machinery. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… claude -p
Two related changes to Step 1, kept in one commit because both touch the same
region of task_orchestration.py and hunk-level staging is unavailable here.
Phase 2 was worse than fail-open; it was fail-silent. call_execution_script
never raises -- it reports failure as a status dict (ERROR / FAILED /
NO_PROGRESS / SCRIPT_NOT_FOUND) -- so the try/except meant to catch a failing
todo_decomposer could not fire, and its warning never logged. Every
script-level failure fell through to `_decomp_raw.get("todo_list", [])`, giving
an empty list with no diagnostic, after which orch_result was written with a
hardcoded "success": True. A decomposer that never ran was indistinguishable
from a task with nothing to do, and the state said the step had succeeded.
The status is now inspected explicitly, failures are logged at ERROR and
collected into orch_result["errors"], and "success" is derived from whether
anything actually failed. This is deliberately not a hard abort, unlike the
missing-template case: the orchestration prompt is still valid here, and in
hook mode the session implements from it, so aborting would break a recoverable
path. The failure is made loud and visible rather than fatal.
prompt_gen_expert_caller no longer spawns `claude -p`. Capturing a generated
prompt from a child's stdout cannot work against the master template, whose
save-and-stop protocol forbids printing it. The script now assembles the prompt
and returns it; llm_response is kept as an empty string so the caller's
existing `llm_response or prompt` routes the assembled prompt downstream with
no contract change. _call_claude_cli, _verify_prompt_schema and the shutil
import went with it.
That removal made three descriptions false, all corrected here: Phase 2's
wrapper text announced "a fully generated orchestration prompt with a
MULTI-AGENT PROMPT BUNDLE" for what is now the template itself; the node
docstring and the PHASE 1 banner both still described a claude CLI subprocess.
STEP1_PROMPT_GEN_SILENCE survives in task_orchestration because layer 1 -- the
python subprocess spawned by call_execution_script -- still exists. Only the
nested claude CLI it also used to bound is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
templates/orchestration_system_prompt.txt has had no reader since Step 1 began resolving ORCHESTRATION_TEMPLATE.md from claude-global-library. Leaving it on disk is the risk: it looks like the template the pipeline uses, and the next reader to open it would find a plausible-looking 198-line file missing nine of the master's steps, STEP 7's mandatory anti-hallucination layer among them. Deleting it removes the only way back to that fork. The templates package went with it -- nothing imports sdlc_pipeline.templates, and the file was its sole content. _TEMPLATES_DIR and _TEMPLATE_FILE are gone from prompt_gen_expert_caller for the same reason. Live documentation is corrected to describe what actually happens now: the master is resolved from the library, and runtime values are prepended as a grounding header rather than substituted into placeholders that the master does not have. Updated in CLAUDE.md's pipeline flow, README.md's Mermaid diagram and directory tree, and PIPELINE_ARCHITECTURE.md's Step 1 node. docs/phase-1-architecture/, docs/releases/ and docs/orchestration_prompt.md still name the old file. Those are point-in-time records of what was true when written, and rewriting them would falsify the history rather than correct it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7ed4a0d deleted sdlc_pipeline/templates/, so the canary tree holds 44 eligible files where it held 45 when the floor was first measured on 2026-08-01. The assertion failed on a deletion that was intentional. Lowering a canary threshold is normally how a real regression gets buried, so the reason is recorded in the docstring rather than left as a bare number change. It is safe here because this test only guards against the tree collapsing to something trivial -- completeness is proven independently by test_canary_symmetric_difference_is_empty, which compares the analysed set against a live enumeration of the tree and would still fail if discovery started missing files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 2 no longer decomposes the orchestration prompt or runs agents. It ran todo_decomposer to split the prompt into a TODO list and orchestrator_agent_caller per TODO, each a nested `claude -p` subprocess. Both re-derived work that belongs elsewhere: the master template's STEP 13 already produces the MULTI-AGENT PROMPT BUNDLE, and this node runs inside a hook subprocess, so it cannot execute in the session it is serving -- it can only hand that session something to run. orchestrator_result now records what was emitted rather than what was executed: mode, prompt_chars, template_source and the claude-global-library VERSION read through the resolver. The version matters because the template lives in the library, so the engine's own version says nothing about what a given trace was built from. It degrades to "unknown" rather than failing -- metadata for traces must never gate a step that has already produced a valid prompt. An empty dict would have satisfied the contracts just as well, but it would have been a false artifact: flow_trace_converter and decision_explainer both read this field to build reports, and handing them a hollow value to keep a postcondition green is how a trace starts lying about what happened. todo_list, todo_results and _lift_todo_agent_output are removed. A prior audit found no reader for any of them outside this module. Also fixes a dormant contract bug found while checking the blast radius. The step0 -> step8 transition guard declared orchestrator_result as a str of at least 50 characters; the node has always written a dict, and node_contracts.py described it correctly as a dict. The two disagreed with each other and one disagreed with reality. It never fired because ENABLE_RUNTIME_VERIFICATION defaults to 0, so the guard set is dormant -- the first operator to enable it would have hit a failure that predates this work entirely. The three scripts (todo_decomposer.py, todo_executor.py, orchestrator_agent_caller.py) are left on disk for now: nothing calls them, but 27 tests in test_todo_decomposition.py and four other test files still cover them, so removing them is a separate decision rather than a side effect of this rewiring. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_guard_step1_to_step2_pass fed a 60-character string for orchestrator_result, matching the spec fb7bbaa corrected. The node has always written a dict, so the test was pinning the bug rather than the behaviour: it would have kept passing no matter how wrong the spec became, because it fed the spec exactly what the spec asked for. The reason it survived is that the guard had only a pass case. A positive test alone cannot detect a wrong type -- it constructs its input from the same mistaken premise. test_guard_step1_to_step2_rejects_string_orchestrator_result supplies the missing half: a str orchestrator_result must be reported as a violation. Verified by reverting the spec to str and confirming the new test fails, then restoring it, so it is known to be capable of failing rather than merely green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
todo_decomposer.py, todo_executor.py and orchestrator_agent_caller.py lost their last caller in fb7bbaa, when Step 1 stopped decomposing and executing the prompt it emits. That commit left them on disk deliberately: 27 tests still covered them, so removing them was a decision to take on its own rather than a side effect of the rewiring. This is that decision. The reason to delete rather than keep is the same one that applied to the forked orchestration template in 7ed4a0d. Dead code that still looks live is worse than absent code, because the next reader finds three plausible scripts describing a nested `claude -p` execution model the pipeline no longer has, and nothing in the files says so. tests/test_todo_decomposition.py goes with them; it imports the deleted modules directly and tests nothing else. Two tests measured the tree these files were in, and both were checked rather than assumed: test_the_named_adr_016_sites_no_longer_carry_a_fixed_timeout named all three in ADR-016's enumeration of 6 timeout sites across 5 files, and asserted the set was readable -- so the deletion would have failed it via `unreadable`. The names are kept rather than dropped. A deleted file trivially carries no fixed timeout, but erasing it from the enumeration also erases the record that the site ever existed, and nothing would then notice if one came back. The tuple is split by what each name now asserts: surviving files are still scanned for fixed timeouts, retired ones are asserted absent. Both halves can still fail. The sdlc_pipeline canary floor moves 44 -> 41. It moved 45 -> 44 in 59f2a41 for the templates purge; this is the same situation and the docstring now records both deletions rather than just the latest. Lowering a canary is normally how a regression gets buried, so it stays safe only because completeness is proven independently by test_canary_symmetric_difference_is_empty, which compares the analysed set against a live enumeration of the tree. Also removes three mock branches for call_execution_script("todo_decomposer") in test_kg_routing.py and test_standards_wiring_fix.py. The node no longer makes that call, so the branches were unreachable -- they would have gone on returning a plausible empty todo_list forever without ever being asked. faithfulness_gate._default_caller's docstring described itself as mirroring _call_claude_cli in prompt_gen_expert_caller, "also duplicated in orchestrator_agent_caller.py and todo_decomposer.py". All three are now gone -- the first was removed in e0d9372 -- so it is described on its own terms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
todo_list, todo_results, completed_todos and current_todo_index were written by todo_decomposer and todo_executor and read by nothing else. Those scripts were deleted in c1d1d47, so the fields now have neither a writer nor a reader anywhere in the engine -- confirmed by a repo-wide grep that returns no hit outside this file. Leaving unwritten fields on FlowState is not free. The struct is the contract a node reads to know what is available to it, and a field that can never be populated is a standing invitation to write a consumer for data that will always be None. completed_todos additionally carried an _merge_lists reducer, so it declared a parallel-write merge policy for a value no node produces. _merge_lists and Annotated are both still used by other fields, so the imports stay. orchestrator_result's comment described it as the "Full result dict from orchestrator-agent-caller", which stopped being true in fb7bbaa when the node began writing it directly. It now says what the field actually holds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…strator_result Both described a node that no longer exists. orchestrator_agent_caller.py was deleted in c1d1d47, and these were its last two traces in the verification layer. verify_orchestrator_result was the more serious of the two. Its signature was `result: str` and its body called .strip(), len() and .startswith() -- but orchestrator_result has always been a dict, and since fb7bbaa it is explicitly a structured emission record. This is the same str-vs-dict mistake that 6982930 found in the step0 -> step8 transition guard, from the same era. It survived for the same reason: its only production caller passed it a raw CLI response string, so nothing ever handed it the value the pipeline actually carries. _MIN_RESULT_LEN and _ERROR_PREFIXES went with it -- both existed solely for that function. ORCHESTRATOR_CONTRACT is removed from NODE_CONTRACT_REGISTRY as well. A contract naming an unrunnable node is not harmlessly dormant: the registry is a node_name -> contract lookup, so it was offering a contract under a key nothing can ever produce. orchestrator.py never imported it (only PRE_ANALYSIS and STEP1), which is why nothing broke. The two e2e tests that used it are repointed rather than deleted. They exercise the verifier's precondition machinery -- register a contract, check a state, assert on violations -- not anything specific to the node named in it. PROMPT_GEN_CONTRACT carries the identical orchestration_prompt >= 200 precondition and names a node that still exists. The second of the pair needed more than a rename. It was "fails with short prompt", and on PROMPT_GEN_CONTRACT that would have been vacuously green: PROMPT_GEN also requires user_message, so a state carrying only a short prompt violates the contract whether or not the length check works at all. It now asserts the missing user_message it actually detects, and the length dimension stays covered by its partner, which asserts no violations for a valid 200+ char prompt. Both halves can fail for the reason they claim. Two descriptions that had become false are corrected in passing: the e2e module docstring still advertised ORCHESTRATOR_CONTRACT as what the file validates, and _map_step1_result_to_state described its orch_result argument as the "Parsed JSON dict returned by orchestrator_agent_caller" -- the node has built that dict itself since fb7bbaa. verify_orchestration_prompt is kept. It also has no production caller, but unlike its sibling it encodes a true contract -- orchestration_prompt genuinely is a str -- so removing it would be a separate judgement rather than part of this cleanup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The live documentation still showed Step 1 decomposing its prompt into TODOs and executing them through nested `claude -p` subprocesses. That stopped being true in fb7bbaa and the three scripts were deleted in c1d1d47, so every diagram and table below described a pipeline the repository no longer contains. CLAUDE.md: the planning-evolution block marked the v1.20 TODO-decomposition pipeline as CURRENT. The v1.20 line is kept -- it is a record of what that version did -- and the CURRENT marker moves to a new 2026-08-07 line, so the history reads as a history rather than being quietly rewritten. The Phase 2 description now explains why the node emits instead of executing: it runs inside a hook subprocess and cannot act in the session it is serving. Three Key Components rows naming deleted files are gone, and the PromptGen row no longer claims a claude CLI call, which e0d9372 removed. CLAUDE.md also advertised STEP1_PROMPT_GEN_TIMEOUT and STEP1_TODO_DECOMPOSER_TIMEOUT. ADR-016 retired both from the code, and test_no_pipeline_module_still_names_a_retired_timeout_variable enforces their absence there -- but nothing enforces it in prose, so the docs went on offering operators two environment variables that do nothing. README.md: the Mermaid Step 1 subgraph called Phase 2 "orchestrator_agent_caller executes full plan, streamed live". The directory tree named the same deleted script. PIPELINE_ARCHITECTURE.md's C2 node described a 30-90s agent run that streams to the terminal; it now states plainly that the phase executes nothing and what orchestrator_result actually holds. CHANGELOG.md gets an [Unreleased] section covering this arc -- the emit-only rewiring, the three deletions, the four dead state fields, the retired contract and verifier, and the dormant str-vs-dict guard bug that predated all of it. Point-in-time records under docs/phase-*/, docs/reports/ and docs/releases/ still name the deleted scripts and are deliberately untouched, the same call 7ed4a0d made: they record what was true when written, and editing them would falsify the history rather than correct it. uml/ is regenerated, not hand-edited. NOT fixed here, and worth a separate decision: CLAUDE.md's Quick Info counts are badly stale independently of this work. It claims 244 langgraph_engine Python files (actual 253), 395 repo-wide (actual 492) and 45 test files (actual 99 -- 90 top-level, 5 integration, 3 e2e, 1 load). README disagrees with CLAUDE.md on the last figure, claiming 44. This deletion moved those numbers by 3 and 1; the rest of the gap long predates the branch, correcting it needs a decision about the counting basis, and folding a ~54-file correction into this commit would bury it. No test asserts any of these numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_a_windows_drive_letter_yields_a_doubled_hyphen failed on both CI runners
(3.10 and 3.11) with:
-home-runner-work-claude-workflow-engine-claude-workflow-engine-C--Users-x
project_slug calls os.path.abspath, and "C:\\Users\\x" is only an absolute path
on Windows. On Linux it is a relative path, so abspath prepends the working
directory and the slug no longer begins with the drive letter. The assertion was
startswith, so it passed on the Windows machine it was written on and could
never pass on the runner. It is the first test on this branch that CI has ever
executed -- the NFR-1 work landed before any PR existed -- which is why nothing
said so until now.
The fix is endswith, not `in`. Both would pass on Linux, but abspath only ever
PREPENDS, so the tail is still the full transform under test on both platforms:
a regression that dropped the colon and produced C-Users-x fails endswith and
would slip past a containment check. Verified three ways: the exact string CI
reported satisfies endswith and not startswith, the Windows slug is C--Users-x
and satisfies endswith, and the mangled C-Users-x is rejected.
The function is not changed. Its docstring documents that project_root is an
absolute path, and abspath there is defensive normalisation; the test was
passing input that only meets that contract on one platform.
The two sibling tests survive on Linux by accident rather than by design -- they
assert with `in` and a "." absence, and the runner's checkout path happens to
contain no dots. Left as they are: they test what they claim, and rewriting
passing tests on a red-CI fix would obscure which change fixed the failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
30 commits, 46 files, +9,980 / -2,431. Two arcs that share a branch because the second was found while proving the first.
Arc 1 -- NFR-1 measurement driver (
1dfc066..13392de)A transcript-tailing observer that can produce a valid NFR-1 measurement. The work is mostly about the two ways the measurement could not fail:
dabcb41,e829a9f), so it closed over the wrong interval.bb94cc5), so a run could be scored before its work finished.pluginmarker was being derived from the plugin root's basename (e177fb0,76d74a3). "Not known" and "shown not to be the plugin" were indistinguishable until7809ca8.b2e5ad6records the first valid measurement and what it took to get one.13392derecords the owner's amendment closing #259 without a PASS -- that is deliberately recorded as an amendment rather than presented as a pass.Arc 2 -- Step 1 stops executing (
0eceeae..226b28c)Step 1 ran
todo_decomposerto split its orchestration prompt into a TODO list, thenorchestrator_agent_calleronce per TODO -- each a nestedclaude -psubprocess. Both re-derived work that belongs elsewhere: the master template's STEP 13 already produces the MULTI-AGENT PROMPT BUNDLE, and the node runs inside a hook subprocess, so it cannot execute in the session it is serving. It can only hand that session something to run.0eceeae,7ed4a0d-- the master template is resolved fromclaude-global-librarywith no fallback; the local 198-line fork (missing nine of the master's steps, including STEP 7's anti-hallucination layer) is deleted.e0d9372-- Phase 2 was fail-silent, not fail-open.call_execution_scriptreports failure as a status dict rather than raising, so thetry/exceptguarding the decomposer could not fire. Every script-level failure fell through to an empty TODO list with no diagnostic, then wrote a hardcoded"success": True. A decomposer that never ran was indistinguishable from a task with nothing to do.fb7bbaa--orchestrator_resultnow records what was emitted (mode,prompt_chars,template_source,library_version) rather than what was executed. An empty dict would satisfy the contracts equally well, butflow_trace_converteranddecision_explainerboth read the field to build reports; a hollow value kept green is how a trace starts lying.c1d1d47,e59a271,94cbaf7-- the three now-callerless scripts, the four unwritten/unreadFlowStatefields, andORCHESTRATOR_CONTRACT+verify_orchestrator_resultare removed.A dormant bug this surfaced
The
step0 -> step8transition guard declaredorchestrator_resultas astrof at least 50 chars. The node has always written adict, andnode_contracts.pydescribed it correctly -- so the two disagreed with each other and one disagreed with reality. It never fired only becauseENABLE_RUNTIME_VERIFICATIONdefaults to0. The first operator to enable verification would have hit a failure predating their change.verify_orchestrator_resultcarried the samestrpremise.6982930is the more interesting half: the guard's only test fed it a 60-character string -- exactly what the wrong spec asked for -- so it would have stayed green no matter how wrong the spec became. A positive test alone cannot detect a wrong type, because it builds its input from the same mistaken premise. It is now paired with a negative case, verified capable of failing by reverting the spec and watching it go red.Tests that measured deleted things
Three tests would have broken on the deletions, and each was resolved rather than loosened:
sdlc_pipelinecanary floor 45 -> 44 -> 41. Safe only because completeness is proven independently bytest_canary_symmetric_difference_is_empty, which compares the analysed set against a live enumeration of the tree.PROMPT_GEN_CONTRACT. One needed more than a rename: "fails with short prompt" would have been vacuously green, sincePROMPT_GENalso requiresuser_messageand a state carrying only a short prompt violates the contract whether or not the length check works. It now asserts the missinguser_messageit actually detects.Verification
Run locally on win32/py3.13 (CI is ubuntu/py3.10-3.11):
secrets_check,verify_policy_audit_matrix,verify_policy_orphan_dispositions,verify_open_encoding,verify_home_paths,verify_no_fixed_timeouts.-m "not integration"suite reaches 100% with zeroF/E. Caveat: the exit-code capture timed out twice at 600s, so this rests on all-dots progress output rather than a green exit code. CI is the authority.runtime_verificationimporter (55 passed / 1 skipped), schema + e2e + transition guards (43), doc governance (9 files), NFR-2 + kg_routing + standards (95).Known drift, deliberately not fixed here
CLAUDE.md's Quick Info counts are stale well beyond this work: it claims 244langgraph_enginePython files (actual 253), 395 repo-wide (actual 492) and 45 test files (actual 99 -- 90 top-level, 5 integration, 3 e2e, 1 load).README.mddisagrees withCLAUDE.mdon the last figure, claiming 44. These deletions moved the numbers by 3 and 1; the rest predates the branch. Correcting it needs a decision about the counting basis, and folding a ~54-file correction in here would bury it. No test asserts these numbers.Point-in-time records under
docs/phase-*/,docs/reports/anddocs/releases/still name the deleted scripts and are deliberately untouched -- they record what was true when written.🤖 Generated with Claude Code