This repository was archived by the owner on Jun 8, 2026. It is now read-only.
FEAT: EVALUATE_PRD command for desktop symphony-loop harness - #30
Merged
Merged
Conversation
- Add EVALUATE_PRD to LoopCommand type and VALID_COMMANDS set - Extract writePrdArtifact shared helper (reused by DECOMPOSE and EVALUATE_PRD) - Add writeArtifactsForEvaluatePrd and readEvaluatePrdOutputs handlers - Extract spawnClaudeFromFile helper to eliminate duplicate spawn logic - Spawn judges:run-judges skill via claude CLI for EVALUATE_PRD command - Add CL_TEST_ALLOW_LOOPBACK_API=1 bypass for validateApiBaseUrl in tests - Export _forTesting with new handlers for unit test access - Add symphony-loop-evaluate-prd.test.ts integration test suite - Bump desktop version to 0.4.4 Testing: new test file covers writePrdArtifact priority logic and readEvaluatePrdOutputs parsing Risks: EVALUATE_PRD uses same temp-dir pattern as DECOMPOSE; cleanup on completion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Code Review SummaryStatus: Changes Requested Reviewers: Bug Hunter A, Bug Hunter B, Unified Auditor, Premise Reviewer, gateway-core-architect Findings
BLOCKING Issues (must fix)
MEDIUM Issues (consider)
Validation Stats
Recommendation: Address the blocking SSRF bypass before merging. The medium items are lower priority but the test false-positive (#2) should be fixed to ensure the cleanup behavior is actually exercised. |
- Replace CL_TEST_ALLOW_LOOPBACK_API env var with _forTesting.overrideValidateApiBaseUrl seam - Collapse writeArtifactsForDecompose and writeArtifactsForEvaluatePrd into shared writePrdArtifact - Export writePrdArtifact and new reset/override helpers via _forTesting Testing: existing test suite updated to use new seam Risks: none — test-only change, no production behaviour altered Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add fs.rm cleanup in handleProcessCompletion when process exits non-zero
- Add fs.rm cleanup in BINARY_NOT_FOUND handler before returning 500
- Prevents orphaned /tmp/symphony-evaluate-prd-* and symphony-decompose-* dirs
Testing: TypeScript typecheck passes (tsc --noEmit clean)
Risks: None — fire-and-forget cleanup, errors swallowed with .catch(() => {})
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e paths - Give each integration test a distinct 8-char loopId prefix so their symphony-evaluate-prd-* work dirs no longer collide; prevents async completion handlers from earlier tests racing against T-5.4's cleanup assertion - Move overrideValidateApiBaseUrl into beforeEach and reset it in afterEach so the SSRF bypass doesn't leak into co-process test suites - Add fs.rm cleanup for EVALUATE_PRD (and DECOMPOSE) temp dirs in the log-open-failure and spawnErr catch blocks, matching the pattern already used for BINARY_NOT_FOUND - Update stale JSDoc on startEventServer that referenced the removed CL_TEST_ALLOW_LOOPBACK_API env var Testing: 242/242 tests pass, typecheck clean Risks: none — test-only changes plus two defensive cleanup lines Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
thadeusb
reviewed
Mar 19, 2026
thadeusb
reviewed
Mar 19, 2026
…_CHANGES PR #30 branched before PRs #31-#33 landed, leaving all three commands on the old spawnClaudeFromFile path (--output-format json, no streaming, no token tracking, no --allowedTools, no --max-turns). This commit brings parity with the fixes merged to main: - Add findStreamFormatter() and buildClaudePipeline() (ported from main) - DECOMPOSE: switch from spawnClaudeFromFile to buildClaudePipeline with -p -, --output-format stream-json, --verbose, --allowedTools, --max-turns 200 - EVALUATE_PRD: same treatment — produces claude-output.jsonl for token tracking, claude-stderr.log for debugging, and readable formatted logs - REQUEST_CHANGES: add -p, --output-format stream-json, --verbose, and route through buildClaudePipeline (was spawning claude directly) - Remove now-dead spawnClaudeFromFile helper Testing: `just desktop-typecheck` clean; all 242 tests pass Risks: None — behaviour change is intentional parity fix; formatter falls back gracefully when stream_formatter.py is not installed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract shared claudeArgs into headlessClaudeArgs constant - Extract spawnFromPromptFile helper used by DECOMPOSE and EVALUATE_PRD - Fix prompt file read racing with fs.rm by reading before waitForEvent - Replace fixed 300ms sleep with polling loop for temp dir cleanup Risks: none — behaviour unchanged, only structural refactor and test stabilisation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use run-judges --workdir runDir; REPO_PATH only when repo linked - Drop CLOSEDLOOP_WORKDIR line from prompt (spawn env unchanged) - Update evaluate-prd tests; bump desktop version Testing: pnpm exec tsx --test test/symphony-loop-evaluate-prd.test.ts Risks: None identified Made-with: Cursor
thadeusb
reviewed
Mar 20, 2026
thadeusb
reviewed
Mar 20, 2026
thadeusb
reviewed
Mar 20, 2026
thadeusb
approved these changes
Mar 20, 2026
shafty023
reviewed
Mar 20, 2026
shafty023
reviewed
Mar 20, 2026
- Add read-json-file-sync.ts; replace duplicated readJsonFile in loop and PRD artifacts - Track usedTempDir so temp claude workdirs are removed on completion and early failures - Bump desktop package version to 0.5.3 Testing: Not run (commit-only) Risks: Low; behavior change is cleanup gating via usedTempDir flag Made-with: Cursor
- Add per-command repo requirement; EVALUATE_PRD treats local repo as optional and logs when localRepoPath or repo.fullName is unusable - Rm decompose/evaluate-prd tmp dirs before mkdir for clean retries - Add tests for stale repo.fullName and disallowed localRepoPath - Bump desktop package version to 0.6.1 Testing: Not run for this commit Risks: EVALUATE_PRD may run without REPO_PATH when repo metadata is wrong; callers relying on hard failure should validate repo client-side Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EVALUATE_PRD command for desktop symphony-loop harness
This PR adds the
EVALUATE_PRDcommand to the desktop symphony-loop harness, enabling PRD evaluation through the local gateway. When invoked, the harness writesprd.mdfrom artifacts (or prompt), spawns the Claude CLI with a prompt that activates thejudges:run-judgesskill, and returnsprd-judges.jsonresults to the cloud control plane.Key Changes
Core Feature:
EVALUATE_PRDcommand insymphony-loop.ts— uses a temp dir (no worktree), writesprd.mdvia sharedwritePrdArtifact, and spawnsclaude -p - --output-format jsonwith an evaluation promptCLOSEDLOOP_WORKDIR, optionalREPO_PATHwhen a repo is linked, and instructs the model to runjudges:run-judges --artifact-type prdreadEvaluatePrdOutputs()to readprd-judges.jsonand return{ prdJudges }in the completed eventSupporting Changes:
writePrdArtifact()for reuse by DECOMPOSE and EVALUATE_PRDspawnClaudeFromFile()to avoid E2BIG when passing large prompts via stdinCL_TEST_ALLOW_LOOPBACK_API=1to bypass private-address checks in testsImpact:
EVALUATE_PRDcommand on/api/engineer/symphony/loop— accepts same body shape as DECOMPOSE (artifacts, prompt, optional repo)