Skip to content

E4: planning acceptance fixtures and recorded Claude output - #45

Merged
mchwang merged 30 commits into
mainfrom
codex/lane-e4-acceptance
Oct 4, 2026
Merged

mchwang merged 30 commits into
mainfrom
codex/lane-e4-acceptance

Conversation

@mchwang

@mchwang mchwang commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Adds E4's planning acceptance: synthetic integration fixtures, and real Claude planning output recorded through lane D and replayed through E2 validation and E3/Store.

Acceptance matrix: docs/implementation/planning-acceptance.md.

Synthetic fixtures (unchanged)

JSON/YAML import, the eight T18 broken-plan cases, stable identity, replay and sibling invalidation, SQLite reopen, malformed responses, hostile prompt input, and exact command approval, composed with the real Store. No synthetic fixture is presented as vendor output.

Recorded Claude output (new)

The #75 blocker is resolved. Codex is refused in every phase (#75, #93), so only Claude is recorded: claude-draft and claude-suggest.

  • runner/planning-provider.ts: E2's provider over lane D. Each request runs in a fresh container on a read-only copy of the head, in the planning phase with vendor-only egress. D passes the schema as --json-schema and returns structured_output. Codex is refused before any Docker or Git work. Planning keeps its own leftovers record, so a planning failure never turns Ask off.
  • scripts/record-planning.ts: a manual, credentialed run. It saves output before validating it, keeps the token out of Docker's environment and printed errors, and prints the runner label if anything is left behind.
  • test/planning-recorded.test.ts: replays the draft into the Store and the suggestion through E3, then reopens SQLite. A recording is stale, and the test fails, when the repository files, prompt, schema, Claude argv, image tag or CLI version change.
  • test/planning-extraction.test.ts: structured envelopes through parseClaudeOutput, then E3 and the Store. Exactly 1 MiB reaches schema validation; 1 MiB + 1 byte fails the size limit. No schema-valid reply is near 1 MiB, because the schema caps every text field.
  • test/planning-provider.test.ts: lane D stand-ins. Covers release ordering, cancellation, attempt and context identity, retained storage, untracked allocations, and planning-worded refusals.

Recorded with codeboost-agent:node26-codex0.153.4-claude2.1.281 (Claude Code 2.1.281, CLI default model; the adapter passes no model flag). Both recordings validate unchanged.

Left for F's wiring

  • The provider's Docker and Git setup calls are synchronous, so a server must run it off its request thread, as Ask does.
  • E3's default 120-second suggestion timeout is shorter than the provider's ten-minute budget.
  • Startup recovery does not yet remove planning's Docker resources after a crash.
  • Image changes that keep the tag and argv, such as Dockerfile packages, do not mark a recording stale.

Validation

Head 64b80b5:

  • npm run typecheck: passed.
  • CI's npm test command (Docker suites excluded): 60 files, 1391 tests passed.
  • git diff --check: passed.
  • An independent subagent review found 19 candidates. Fixed: recording staleness (files and argv hashes), token handling in the script, vacuous reasons, and missing path tests.
  • Three /code-review high rounds:
    • Round 1 (9 findings, 7 fixed): the original error now wins over cleanup errors; RetainedStorage takes a LeftoverPolicy; exact reasons in the T18 and malformed-response cases; fileURLToPath.
    • Round 2 (4 findings, all fixed): a PlanningStorage type keeps Ask's instance out; removePlanningRoot deletes only planning roots and makes the input writable first; a failed request reports both its error and the cleanup failure.
    • Round 3: no findings.
  • Skipped, and left for F's wiring (see the doc): merging the provider with askInContainer, and covering lane D's host staging with a planning TMPDIR.

🤖 Generated with Claude Code

# Conflicts:
#	core/planning-suggestions.ts
#	docs/implementation/planning-suggestions.md
#	test/planning-suggestions.test.ts
@mchwang

mchwang commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot review

@mchwang
mchwang marked this pull request as ready for review September 24, 2026 18:26
@mchwang

mchwang commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Real authoring-output validation remains blocked on the D5 provider and settlement gate.

Review effort: Lite
Findings: None

What changed in this PR

Adds synthetic planning acceptance fixtures and Store-backed integration tests, while documenting the remaining D5 live-provider dependency.

Changes:

  • Added acceptance tests for imports, persistence, identity, replay, invalidation, malformed responses, hostile input, and command approval.
  • Added synthetic plan, edit, and hostile-input fixtures.
  • Documented acceptance coverage and deferred real-provider validation.
File Description
test/​planning-acceptance.test.ts Integrated planning acceptance tests
test/​fixtures/​planning/​synthetic-plan.json Synthetic plan fixture
test/​fixtures/​planning/​synthetic-edits.json Synthetic suggestion fixture
test/​fixtures/​planning/​hostile-input.json Hostile-input fixture
docs/​implementation/​planning-acceptance.md Acceptance matrix and D5 dependency

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@mchwang

mchwang commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Blocked by #75. The real Claude and Codex planning recordings, captured through D's production planning path, all fail E2 validation. Neither vendor receives the answer schema, and Codex cannot read files in read-only phases. The recording provider, capture script and replay tests are ready but not yet pushed. After #75 lands, I will record again, push, and update this PR's evidence.

🤖 Generated with Claude Code

mchwang and others added 2 commits October 3, 2026 15:20
Add the planning provider over lane D's container (read-only planning phase,
vendor-only egress, Claude only), a credentialed recording script, and replay
tests that drive real Claude output through E2 validation and E3/Store.

Codex is refused in every phase (#75, #93), so two recordings are kept:
claude-draft and claude-suggest. Each recording stores hashes of the
repository files, prompt, schema and the exact Claude argv, so a lane D or
prompt change makes it stale. Extraction tests follow the structured
(--json-schema / structured_output) envelope. Planning keeps its own record of
leftovers, so its refusals never turn Ask off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mchwang mchwang changed the title E4: add planning acceptance fixtures; await D5 recordings E4: planning acceptance fixtures and recorded Claude output Oct 3, 2026
mchwang and others added 2 commits October 3, 2026 16:17
- Keep a failed request's own error when its cleanup also fails; leftovers are
  retained and the next request names them.
- RetainedStorage takes a LeftoverPolicy (refusal wording and host-copy
  removal); planning uses PLANNING_LEFTOVERS instead of a wrapper that
  swallowed RetainedStorage's errors.
- Assert exact reasons in the T18 broken-plan and malformed-response cases,
  test each planning refusal, and drop an assertion that could not fail.
- Use fileURLToPath for fixture directories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- PlanningStorage types planning's leftovers apart from Ask's, so an
  Ask-policy RetainedStorage cannot be passed to the planning provider.
- removePlanningRoot deletes only planning roots under the temporary
  directory, and makes the read-only input directory writable first, also on
  retries.
- A failed request reports its own error followed by the cleanup failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mchwang
mchwang marked this pull request as ready for review October 4, 2026 00:55
@mchwang
mchwang merged commit fe4155e into main Oct 4, 2026
3 checks passed
@mchwang
mchwang deleted the codex/lane-e4-acceptance branch October 4, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants