Repository navigation
Run verify_orchestration_prompt on what Step 1 emits - #299
Merged
Merged
Conversation
…mits The function was written, exported from runtime_verification and covered by four unit tests, and called by nothing. Its sibling verify_orchestrator_result was deleted in 94cbaf7 for that reason plus a false premise -- a str signature for a value that has always been a dict. This one encodes a true contract, so the choice was to wire it or delete it, and leaving it was the only option that got worse over time: a correct-but-uncalled validator looks like coverage that does not exist. Closes item 4 of #296. The gap it closes is narrower than "the prompt is unchecked", and checking first changed the justification. Both degraded paths in the node ALREADY log -- an ERROR from prompt_gen falls back to the raw task at logger.error, and an empty llm_response/prompt falls back at logger.warning. So this is not here to make those visible. It is here for the case neither covers: a response that is non-empty, and therefore silent, but not a usable orchestration prompt -- a truncated template, the wrong file, a stub. Until now that was indistinguishable from a good prompt. Warnings are recorded on orchestrator_result["prompt_warnings"] and logged, and are never fatal. STEP1_CONTRACT documents the short raw-task fallback as a legitimate degraded path, and that fallback trips both of the verifier's checks by design, so failing the step here would turn every recoverable run into a dead one. The same reasoning e0d9372 applied to Phase 2: loud and visible rather than fatal. Wiring was confirmed possible before it was done rather than after: the master template resolved from claude-global-library is 1,996 lines and 335 KB and contains "Phase" 386 times, so the happy path clears both the 200-character floor and the keyword check without a spurious warning on every run. The new test drives step1_task_analysis_node rather than calling the verifier directly, because tests/test_schema_verifier.py already covers the function in isolation and that coverage is exactly what failed to notice it had no callers. The two cases are a pair on purpose: "a good prompt produces no warnings" passes just as well when the verifier is never invoked, since an unwired check and a satisfied check both return an empty list. Proven by replacing the call with a literal [] and confirming the malformed-prompt case and the not-fatal case both go red while the well-formed case stayed green, then restoring it. Counts move to 489 / 101 / 92 / 122. Caught by tests/test_doc_counts_match_the_tree.py across all seven claim sites rather than by hand -- the second time that enforcement has paid for itself since it landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes item 4 of #296, the last of the four.
verify_orchestration_prompt()was written, exported fromruntime_verificationand covered by four unit tests — and called by nothing. Its siblingverify_orchestrator_result()was deleted in94cbaf7for that reason plus a false premise (astrsignature for a value that has always been adict). This one encodes a true contract, so the choice was wire it or delete it. Leaving it was the only option that got worse over time: a correct-but-uncalled validator looks like coverage that does not exist.Checking first changed the justification
I expected the argument to be "the emitted prompt is unchecked". That was half true and the wrong half. Both degraded paths in the node already log — an
ERRORfrom prompt-gen falls back to the raw task atlogger.error, and an emptyllm_response/promptfalls back atlogger.warning.The real gap is narrower and, I think, more worth closing: a response that is non-empty, and therefore silent, but not a usable orchestration prompt — a truncated template, the wrong file, a stub. Until now that was indistinguishable from a good prompt.
Not fatal, deliberately
Findings land on
orchestrator_result["prompt_warnings"]and are logged. They never fail the step.STEP1_CONTRACTdocuments the short raw-task fallback as a legitimate degraded path, and that fallback trips both of the verifier's checks by design. Failing here would convert every recoverable run into a dead one — the same reasoninge0d9372applied to Phase 2: loud and visible rather than fatal.Wiring was confirmed possible before it was done
The master template resolved from
claude-global-libraryis 1,996 lines / 335 KB and contains"Phase"386 times, so the happy path clears both the 200-character floor and the keyword check. Without that check first, this could have logged a spurious warning on every single run.The test drives the node, not the function
tests/test_schema_verifier.pyalready covers the function in isolation — and that coverage is exactly what failed to notice it had no callers. So every assertion here goes throughstep1_task_analysis_node.The two cases are a pair on purpose. "A good prompt produces no warnings" passes just as well when the verifier is never invoked, because an unwired check and a satisfied check both return an empty list.
Proven by unwiring it: replacing the call with a literal
[]turned the malformed-prompt case and the not-fatal case red while the well-formed case stayed green. Then restored. The pair can fail for the reason it claims.Counts
489 / 101 / 92 / 122 — caught across all seven claim sites by
tests/test_doc_counts_match_the_tree.pyrather than by hand. Second time that enforcement has paid for itself since it landed two PRs ago.Verified
78 targeted tests (doc counts, docs index, this wire-in, schema verifier, kg-routing, standards wiring, transition guards), all six CI gates, clean collection, no line-ending flips.
ruffis not on PATH in this shell so the lint gate was left to pre-commit and CI, as with #298 where it passed.🤖 Generated with Claude Code