Skip to content

Run verify_orchestration_prompt on what Step 1 emits - #299

Merged
piyushmakhija28 merged 1 commit into
mainfrom
feat/wire-orchestration-prompt-verifier
Aug 7, 2026
Merged

piyushmakhija28 merged 1 commit into
mainfrom
feat/wire-orchestration-prompt-verifier

Conversation

@piyushmakhija28

Copy link
Copy Markdown
Collaborator

Closes item 4 of #296, the last of the four.

verify_orchestration_prompt() was written, exported from runtime_verification and covered by four unit tests — and called by nothing. Its sibling verify_orchestrator_result() was deleted in 94cbaf7 for that reason plus a false premise (a str signature for a value that has always been a dict). This one encodes a true contract, so the choice was wire it or delete it. Leaving it was the only option that got worse over time: a correct-but-uncalled validator looks like coverage that does not exist.

Checking first changed the justification

I expected the argument to be "the emitted prompt is unchecked". That was half true and the wrong half. Both degraded paths in the node already log — an ERROR from prompt-gen falls back to the raw task at logger.error, and an empty llm_response/prompt falls back at logger.warning.

The real gap is narrower and, I think, more worth closing: a response that is non-empty, and therefore silent, but not a usable orchestration prompt — a truncated template, the wrong file, a stub. Until now that was indistinguishable from a good prompt.

Not fatal, deliberately

Findings land on orchestrator_result["prompt_warnings"] and are logged. They never fail the step.

STEP1_CONTRACT documents the short raw-task fallback as a legitimate degraded path, and that fallback trips both of the verifier's checks by design. Failing here would convert every recoverable run into a dead one — the same reasoning e0d9372 applied to Phase 2: loud and visible rather than fatal.

Wiring was confirmed possible before it was done

The master template resolved from claude-global-library is 1,996 lines / 335 KB and contains "Phase" 386 times, so the happy path clears both the 200-character floor and the keyword check. Without that check first, this could have logged a spurious warning on every single run.

The test drives the node, not the function

tests/test_schema_verifier.py already covers the function in isolation — and that coverage is exactly what failed to notice it had no callers. So every assertion here goes through step1_task_analysis_node.

The two cases are a pair on purpose. "A good prompt produces no warnings" passes just as well when the verifier is never invoked, because an unwired check and a satisfied check both return an empty list.

Proven by unwiring it: replacing the call with a literal [] turned the malformed-prompt case and the not-fatal case red while the well-formed case stayed green. Then restored. The pair can fail for the reason it claims.

Counts

489 / 101 / 92 / 122 — caught across all seven claim sites by tests/test_doc_counts_match_the_tree.py rather than by hand. Second time that enforcement has paid for itself since it landed two PRs ago.

Verified

78 targeted tests (doc counts, docs index, this wire-in, schema verifier, kg-routing, standards wiring, transition guards), all six CI gates, clean collection, no line-ending flips. ruff is not on PATH in this shell so the lint gate was left to pre-commit and CI, as with #298 where it passed.

🤖 Generated with Claude Code

…mits

The function was written, exported from runtime_verification and covered by four
unit tests, and called by nothing. Its sibling verify_orchestrator_result was
deleted in 94cbaf7 for that reason plus a false premise -- a str signature for a
value that has always been a dict. This one encodes a true contract, so the
choice was to wire it or delete it, and leaving it was the only option that got
worse over time: a correct-but-uncalled validator looks like coverage that does
not exist. Closes item 4 of #296.

The gap it closes is narrower than "the prompt is unchecked", and checking first
changed the justification. Both degraded paths in the node ALREADY log -- an
ERROR from prompt_gen falls back to the raw task at logger.error, and an empty
llm_response/prompt falls back at logger.warning. So this is not here to make
those visible. It is here for the case neither covers: a response that is
non-empty, and therefore silent, but not a usable orchestration prompt -- a
truncated template, the wrong file, a stub. Until now that was indistinguishable
from a good prompt.

Warnings are recorded on orchestrator_result["prompt_warnings"] and logged, and
are never fatal. STEP1_CONTRACT documents the short raw-task fallback as a
legitimate degraded path, and that fallback trips both of the verifier's checks
by design, so failing the step here would turn every recoverable run into a dead
one. The same reasoning e0d9372 applied to Phase 2: loud and visible rather than
fatal.

Wiring was confirmed possible before it was done rather than after: the master
template resolved from claude-global-library is 1,996 lines and 335 KB and
contains "Phase" 386 times, so the happy path clears both the 200-character
floor and the keyword check without a spurious warning on every run.

The new test drives step1_task_analysis_node rather than calling the verifier
directly, because tests/test_schema_verifier.py already covers the function in
isolation and that coverage is exactly what failed to notice it had no callers.
The two cases are a pair on purpose: "a good prompt produces no warnings" passes
just as well when the verifier is never invoked, since an unwired check and a
satisfied check both return an empty list. Proven by replacing the call with a
literal [] and confirming the malformed-prompt case and the not-fatal case both
go red while the well-formed case stayed green, then restoring it.

Counts move to 489 / 101 / 92 / 122. Caught by
tests/test_doc_counts_match_the_tree.py across all seven claim sites rather than
by hand -- the second time that enforcement has paid for itself since it landed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@piyushmakhija28
piyushmakhija28 merged commit 38e4572 into main Aug 7, 2026
6 checks passed
@piyushmakhija28
piyushmakhija28 deleted the feat/wire-orchestration-prompt-verifier branch August 7, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant