Skip to content
This repository was archived by the owner on Jun 8, 2026. It is now read-only.

FEAT: EVALUATE_PRD command for desktop symphony-loop harness - #30

Merged
aponamarev merged 13 commits into
mainfrom
feat/evaluate-prd-desktop-harness
Mar 20, 2026
Merged

aponamarev merged 13 commits into
mainfrom
feat/evaluate-prd-desktop-harness

Conversation

@aponamarev

@aponamarev aponamarev commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

EVALUATE_PRD command for desktop symphony-loop harness

This PR adds the EVALUATE_PRD command to the desktop symphony-loop harness, enabling PRD evaluation through the local gateway. When invoked, the harness writes prd.md from artifacts (or prompt), spawns the Claude CLI with a prompt that activates the judges:run-judges skill, and returns prd-judges.json results to the cloud control plane.

Key Changes

Core Feature:

  • Implemented EVALUATE_PRD command in symphony-loop.ts — uses a temp dir (no worktree), writes prd.md via shared writePrdArtifact, and spawns claude -p - --output-format json with an evaluation prompt
  • Prompt includes CLOSEDLOOP_WORKDIR, optional REPO_PATH when a repo is linked, and instructs the model to run judges:run-judges --artifact-type prd
  • Added readEvaluatePrdOutputs() to read prd-judges.json and return { prdJudges } in the completed event
  • Temp dir cleanup after completion (same pattern as DECOMPOSE)

Supporting Changes:

  • Extracted writePrdArtifact() for reuse by DECOMPOSE and EVALUATE_PRD
  • Extracted spawnClaudeFromFile() to avoid E2BIG when passing large prompts via stdin
  • Added CL_TEST_ALLOW_LOOPBACK_API=1 to bypass private-address checks in tests
  • Comprehensive test suite for EVALUATE_PRD (prompt contents, repo/no-repo, output parsing, temp cleanup, BINARY_NOT_FOUND)

Impact:

  • API: New EVALUATE_PRD command on /api/engineer/symphony/loop — accepts same body shape as DECOMPOSE (artifacts, prompt, optional repo)

- Add EVALUATE_PRD to LoopCommand type and VALID_COMMANDS set
- Extract writePrdArtifact shared helper (reused by DECOMPOSE and EVALUATE_PRD)
- Add writeArtifactsForEvaluatePrd and readEvaluatePrdOutputs handlers
- Extract spawnClaudeFromFile helper to eliminate duplicate spawn logic
- Spawn judges:run-judges skill via claude CLI for EVALUATE_PRD command
- Add CL_TEST_ALLOW_LOOPBACK_API=1 bypass for validateApiBaseUrl in tests
- Export _forTesting with new handlers for unit test access
- Add symphony-loop-evaluate-prd.test.ts integration test suite
- Bump desktop version to 0.4.4

Testing: new test file covers writePrdArtifact priority logic and readEvaluatePrdOutputs parsing
Risks: EVALUATE_PRD uses same temp-dir pattern as DECOMPOSE; cleanup on completion

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@aponamarev aponamarev changed the title FEAT-73: add EVALUATE_PRD command to desktop symphony-loop harness FEAT: EVALUATE_PRD command for desktop symphony-loop harness Mar 18, 2026
@aponamarev
aponamarev requested a review from thadeusb March 18, 2026 15:06
Comment thread apps/desktop/src/server/operations/symphony-loop.ts Outdated
Comment thread apps/desktop/src/server/operations/symphony-loop.ts Outdated
Comment thread apps/desktop/test/symphony-loop-evaluate-prd.test.ts Outdated
Comment thread apps/desktop/src/server/operations/symphony-loop.ts Outdated
@closedloop-ai-stage

Copy link
Copy Markdown

Code Review Summary

Status: Changes Requested

Reviewers: Bug Hunter A, Bug Hunter B, Unified Auditor, Premise Reviewer, gateway-core-architect

Findings

Severity Count
Blocking 1
High 0
Medium 3

BLOCKING Issues (must fix)

  1. [P0] [symphony-loop.ts:126] Test-only SSRF bypass env var baked into production validation code — CL_TEST_ALLOW_LOOPBACK_API=1 disables all SSRF protections in validateApiBaseUrl() with no app.isPackaged guard; if this env var leaks into a production environment the gateway will forward requests to loopback/private addresses including cloud metadata endpoints.

MEDIUM Issues (consider)

  1. [P2] [symphony-loop.ts:511] writeArtifactsForEvaluatePrd is a 100%-identical pass-through wrapper of writeArtifactsForDecompose; both bodies are await writePrdArtifact(...). The shared helper writePrdArtifact already exists for this purpose — the extra wrapper adds no value.
  2. [P2] [symphony-loop-evaluate-prd.test.ts:479] T-5.4 cleanup assertion is a false positive: all integration test loopIds share the same first 8 chars (00000000), so they all write to the same OS temp dir. T-5.4's cleanup assertion passes because T-5.2 already triggered cleanup — not because T-5.4's own code path works.
  3. [P2] [symphony-loop.ts:861] EVALUATE_PRD temp dir is only cleaned up on the success path (exit code 0). Non-zero claude exit and BINARY_NOT_FOUND both leave the symphony-evaluate-prd-<loopId> directory behind.

Validation Stats

  • Total findings from agents: 5
  • Duplicates merged: 1
  • Cross-file grouped: 0

Recommendation: Address the blocking SSRF bypass before merging. The medium items are lower priority but the test false-positive (#2) should be fixed to ensure the cleanup behavior is actually exercised.

aponamarev and others added 4 commits March 18, 2026 10:34
- Replace CL_TEST_ALLOW_LOOPBACK_API env var with _forTesting.overrideValidateApiBaseUrl seam
- Collapse writeArtifactsForDecompose and writeArtifactsForEvaluatePrd into shared writePrdArtifact
- Export writePrdArtifact and new reset/override helpers via _forTesting

Testing: existing test suite updated to use new seam
Risks: none — test-only change, no production behaviour altered

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add fs.rm cleanup in handleProcessCompletion when process exits non-zero
- Add fs.rm cleanup in BINARY_NOT_FOUND handler before returning 500
- Prevents orphaned /tmp/symphony-evaluate-prd-* and symphony-decompose-* dirs

Testing: TypeScript typecheck passes (tsc --noEmit clean)
Risks: None — fire-and-forget cleanup, errors swallowed with .catch(() => {})

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e paths

- Give each integration test a distinct 8-char loopId prefix so their
  symphony-evaluate-prd-* work dirs no longer collide; prevents async
  completion handlers from earlier tests racing against T-5.4's cleanup
  assertion
- Move overrideValidateApiBaseUrl into beforeEach and reset it in
  afterEach so the SSRF bypass doesn't leak into co-process test suites
- Add fs.rm cleanup for EVALUATE_PRD (and DECOMPOSE) temp dirs in the
  log-open-failure and spawnErr catch blocks, matching the pattern
  already used for BINARY_NOT_FOUND
- Update stale JSDoc on startEventServer that referenced the removed
  CL_TEST_ALLOW_LOOPBACK_API env var

Testing: 242/242 tests pass, typecheck clean

Risks: none — test-only changes plus two defensive cleanup lines

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread apps/desktop/src/server/operations/symphony-loop.ts Outdated
Comment thread apps/desktop/src/server/operations/symphony-loop.ts Outdated
aponamarev and others added 5 commits March 19, 2026 16:53
…_CHANGES

PR #30 branched before PRs #31-#33 landed, leaving all three commands on
the old spawnClaudeFromFile path (--output-format json, no streaming, no
token tracking, no --allowedTools, no --max-turns). This commit brings
parity with the fixes merged to main:

- Add findStreamFormatter() and buildClaudePipeline() (ported from main)
- DECOMPOSE: switch from spawnClaudeFromFile to buildClaudePipeline with
  -p -, --output-format stream-json, --verbose, --allowedTools, --max-turns 200
- EVALUATE_PRD: same treatment — produces claude-output.jsonl for token
  tracking, claude-stderr.log for debugging, and readable formatted logs
- REQUEST_CHANGES: add -p, --output-format stream-json, --verbose, and
  route through buildClaudePipeline (was spawning claude directly)
- Remove now-dead spawnClaudeFromFile helper

Testing: `just desktop-typecheck` clean; all 242 tests pass
Risks: None — behaviour change is intentional parity fix; formatter falls
       back gracefully when stream_formatter.py is not installed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract shared claudeArgs into headlessClaudeArgs constant
- Extract spawnFromPromptFile helper used by DECOMPOSE and EVALUATE_PRD
- Fix prompt file read racing with fs.rm by reading before waitForEvent
- Replace fixed 300ms sleep with polling loop for temp dir cleanup

Risks: none — behaviour unchanged, only structural refactor and test stabilisation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use run-judges --workdir runDir; REPO_PATH only when repo linked
- Drop CLOSEDLOOP_WORKDIR line from prompt (spawn env unchanged)
- Update evaluate-prd tests; bump desktop version

Testing: pnpm exec tsx --test test/symphony-loop-evaluate-prd.test.ts
Risks: None identified
Made-with: Cursor
@aponamarev
aponamarev requested a review from thadeusb March 20, 2026 18:52
Comment thread apps/desktop/src/server/operations/symphony-prd-artifacts.ts Outdated
Comment thread apps/desktop/src/server/operations/symphony-loop.ts
Comment thread apps/desktop/src/server/operations/symphony-loop.ts Outdated
Comment thread apps/desktop/src/server/operations/symphony-loop.ts
Comment thread apps/desktop/src/server/operations/symphony-loop.ts
- Add read-json-file-sync.ts; replace duplicated readJsonFile in loop and PRD artifacts
- Track usedTempDir so temp claude workdirs are removed on completion and early failures
- Bump desktop package version to 0.5.3

Testing: Not run (commit-only)

Risks: Low; behavior change is cleanup gating via usedTempDir flag
Made-with: Cursor
- Add per-command repo requirement; EVALUATE_PRD treats local repo as
  optional and logs when localRepoPath or repo.fullName is unusable
- Rm decompose/evaluate-prd tmp dirs before mkdir for clean retries
- Add tests for stale repo.fullName and disallowed localRepoPath
- Bump desktop package version to 0.6.1

Testing: Not run for this commit
Risks: EVALUATE_PRD may run without REPO_PATH when repo metadata is
  wrong; callers relying on hard failure should validate repo client-side
Made-with: Cursor
@aponamarev
aponamarev merged commit d7e3de5 into main Mar 20, 2026
2 checks passed
@aponamarev
aponamarev deleted the feat/evaluate-prd-desktop-harness branch March 20, 2026 22:28
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants