Repository navigation
E4: planning acceptance fixtures and recorded Claude output - #45
Conversation
# Conflicts: # core/planning-suggestions.ts # docs/implementation/planning-suggestions.md # test/planning-suggestions.test.ts
|
@copilot review |
|
@copilot review |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Real authoring-output validation remains blocked on the D5 provider and settlement gate.
Review effort: Lite
Findings: None
What changed in this PR
Adds synthetic planning acceptance fixtures and Store-backed integration tests, while documenting the remaining D5 live-provider dependency.
Changes:
- Added acceptance tests for imports, persistence, identity, replay, invalidation, malformed responses, hostile input, and command approval.
- Added synthetic plan, edit, and hostile-input fixtures.
- Documented acceptance coverage and deferred real-provider validation.
| File | Description |
|---|---|
test/planning-acceptance.test.ts |
Integrated planning acceptance tests |
test/fixtures/planning/synthetic-plan.json |
Synthetic plan fixture |
test/fixtures/planning/synthetic-edits.json |
Synthetic suggestion fixture |
test/fixtures/planning/hostile-input.json |
Hostile-input fixture |
docs/implementation/planning-acceptance.md |
Acceptance matrix and D5 dependency |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Blocked by #75. The real Claude and Codex planning recordings, captured through D's production planning path, all fail E2 validation. Neither vendor receives the answer schema, and Codex cannot read files in read-only phases. The recording provider, capture script and replay tests are ready but not yet pushed. After #75 lands, I will record again, push, and update this PR's evidence. 🤖 Generated with Claude Code |
Add the planning provider over lane D's container (read-only planning phase, vendor-only egress, Claude only), a credentialed recording script, and replay tests that drive real Claude output through E2 validation and E3/Store. Codex is refused in every phase (#75, #93), so two recordings are kept: claude-draft and claude-suggest. Each recording stores hashes of the repository files, prompt, schema and the exact Claude argv, so a lane D or prompt change makes it stale. Extraction tests follow the structured (--json-schema / structured_output) envelope. Planning keeps its own record of leftovers, so its refusals never turn Ask off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep a failed request's own error when its cleanup also fails; leftovers are retained and the next request names them. - RetainedStorage takes a LeftoverPolicy (refusal wording and host-copy removal); planning uses PLANNING_LEFTOVERS instead of a wrapper that swallowed RetainedStorage's errors. - Assert exact reasons in the T18 broken-plan and malformed-response cases, test each planning refusal, and drop an assertion that could not fail. - Use fileURLToPath for fixture directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- PlanningStorage types planning's leftovers apart from Ask's, so an Ask-policy RetainedStorage cannot be passed to the planning provider. - removePlanningRoot deletes only planning roots under the temporary directory, and makes the read-only input directory writable first, also on retries. - A failed request reports its own error followed by the cleanup failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds E4's planning acceptance: synthetic integration fixtures, and real Claude planning output recorded through lane D and replayed through E2 validation and E3/Store.
Acceptance matrix:
docs/implementation/planning-acceptance.md.Synthetic fixtures (unchanged)
JSON/YAML import, the eight T18 broken-plan cases, stable identity, replay and sibling invalidation, SQLite reopen, malformed responses, hostile prompt input, and exact command approval, composed with the real Store. No synthetic fixture is presented as vendor output.
Recorded Claude output (new)
The #75 blocker is resolved. Codex is refused in every phase (#75, #93), so only Claude is recorded:
claude-draftandclaude-suggest.runner/planning-provider.ts: E2's provider over lane D. Each request runs in a fresh container on a read-only copy of the head, in the planning phase with vendor-only egress. D passes the schema as--json-schemaand returnsstructured_output. Codex is refused before any Docker or Git work. Planning keeps its own leftovers record, so a planning failure never turns Ask off.scripts/record-planning.ts: a manual, credentialed run. It saves output before validating it, keeps the token out of Docker's environment and printed errors, and prints the runner label if anything is left behind.test/planning-recorded.test.ts: replays the draft into the Store and the suggestion through E3, then reopens SQLite. A recording is stale, and the test fails, when the repository files, prompt, schema, Claude argv, image tag or CLI version change.test/planning-extraction.test.ts: structured envelopes throughparseClaudeOutput, then E3 and the Store. Exactly 1 MiB reaches schema validation; 1 MiB + 1 byte fails the size limit. No schema-valid reply is near 1 MiB, because the schema caps every text field.test/planning-provider.test.ts: lane D stand-ins. Covers release ordering, cancellation, attempt and context identity, retained storage, untracked allocations, and planning-worded refusals.Recorded with
codeboost-agent:node26-codex0.153.4-claude2.1.281(Claude Code 2.1.281, CLI default model; the adapter passes no model flag). Both recordings validate unchanged.Left for F's wiring
Validation
Head
64b80b5:npm run typecheck: passed.npm testcommand (Docker suites excluded): 60 files, 1391 tests passed.git diff --check: passed./code-review highrounds:fileURLToPath.PlanningStoragetype keeps Ask's instance out;removePlanningRootdeletes only planning roots and makes the input writable first; a failed request reports both its error and the cleanup failure.askInContainer, and covering lane D's host staging with a planning TMPDIR.🤖 Generated with Claude Code