fix(ce-plan): show the scoping checkpoint before writing a brainstorm-sourced plan (EveryInc#1817)

Author: tmchow

skills/ce-plan/SKILL.md

@@ -46,13 +46,13 @@ Phases run in order unless a reference routes out or short-circuits. Read a phas
 
 ### Phases 1-4: Research and Compose
 
-4. Read `references/research.md` before gathering context. It defines local and external research, consolidation, depth reclassification, flow analysis, and the Bake-off gate.
-5. Read `references/structure.md` before resolving questions or structuring the plan. It defines settled-decision handling, stable U-IDs, technical design, depth, and planning boundaries.
+4. Read `references/research.md` before gathering context.
+5. Read `references/structure.md` before resolving questions or structuring the plan.
 6. Compose from `references/plan-sections.md` plus the format-rendering reference selected by `output-mode.md`.
 
 ### Phase 5: Review, Write, Deepen, and Hand Off
 
-7. Read `references/final-review.md` before the pre-write review. It defines Phase 5.1 through 5.3.2: scoping synthesis, writing the file, unified-plan metadata, confidence mode, and deepening.
+7. Read `references/final-review.md` before the pre-write review; it defines Phase 5.1 through 5.3.2. On an interactive run from a brainstorm, the user sees the Phase 5.1.5 scoping synthesis in chat before the write: a confirmation to wait on, or its one-line auto-proceed announcement.
 8. **Model elevation.** Immediately before authoring, read `references/reasoning-elevation.md`, resolve the choice at this boundary, and follow it. Do not author until activation resolution has completed and any selected dispatch or transparent fallback has settled.
 9. In pipeline mode, evidence that invalidates a decision settled earlier in the session stops the write. Return the exact token `settled-decision-invalidated`, the decision, and the reason; do not resolve it silently.
 10. Write the plan before presenting options, then complete the confidence path `final-review.md` defines.

skills/ce-plan/references/final-review.md

@@ -45,12 +45,14 @@ Run this step **whenever Phase 0.2 resolved an upstream Product Contract source*
 
 **Required output — do not skip; silent proceeding is not allowed.** Compose an internal three-bucket scope draft (Stated / Inferred / Out of scope — internal thinking that feeds plan-body routing at Phase 5.2, not the chat output). Derive call-outs (specific forks where user input materially changes the plan), then emit the **brainstorm-sourced** synthesis and **wait for user confirmation before continuing to Phase 5.2.** Its summary is two parts — a one-sentence anchor naming the brainstorm's scope in the brainstorm's own vocabulary, then the plan-specific scoping decisions the brainstorm did not make (full-brainstorm coverage vs. narrowed subset; adjacent refactors in or out; test scope at scenario level) — each affirmable without reading code, and never an enumeration of Implementation Units, file paths, or PR/sequencing shape. Use the confirmation or auto-proceed lines in `references/synthesis-summary.md` (loaded above) rather than reconstructing them here.
 
-**Blocking decision:** auto-proceed — announce without waiting — only when plan depth is **Lightweight AND zero call-outs survive**. Standard and Deep always ask for confirmation, even with zero call-outs.
+**Blocking decision:** auto-proceed — send the user the auto-proceed announcement in chat, then continue without waiting — only when plan depth is **Lightweight AND zero call-outs survive**. Standard and Deep always ask for confirmation, even with zero call-outs.
 
 **Headless / opt-in skip:** in headless mode, or when `SKIP_SCOPING_CONFIRM` resolved to skip in Phase 0.0, do not block — compose the internal draft, skip the chat-time confirmation, and route Inferred bets to a `## Assumptions` section at plan-write (Phase 5.2). The skip covers only this scoping confirmation; Phase 0.4 routing, Phase 0.5 blockers, Phase 2 questions, source-doc disambiguation, and the Phase 5.4 menu still run. Announcement wording and full routing: `references/synthesis-summary.md` ("Headless mode", "When to skip the blocking confirmation").
 
 #### 5.2 Write Plan File
 
+**Before the write, on an interactive run from an upstream Product Contract, check the chat:** the user must already have seen this run's Phase 5.1.5 synthesis or its auto-proceed announcement. Answers to your own planning questions are not that checkpoint, because the synthesis is where those answers show up as scope. If neither is in chat, send it now.
+
 Return to SKILL.md for its Model elevation step (Phase 5 workflow item 8), which decides whether a different model authors the plan. Resume here only after that step has finished; this reference does not dispatch the authoring route itself.
 
 **REQUIRED: Write the plan file to disk before presenting any options.**

skills/ce-plan/references/intake.md

@@ -153,6 +153,6 @@ Runs **only in solo invocation** — when Phase 0.2 found no upstream Product Co
 
 **Required output of this step — do not skip; silent proceeding is not allowed.** Compose an internal three-bucket scope draft (Stated / Inferred / Out of scope — internal thinking that feeds where content goes in the plan body at Phase 5.2, not the chat output). Derive call-outs (specific forks where user input materially changes the plan), then emit the **solo-variant** synthesis and **wait for user confirmation before continuing to Phase 1.** It states the problem as you understand it, then a scope claim — what the plan will target, what it will not, at affirm-or-redirect level — never an enumeration of Implementation Units, file paths, or PR/sequencing shape (the plan-write step decides those, and they are not knowable yet). Use the confirmation or auto-proceed lines in `references/synthesis-summary.md` (loaded above) rather than reconstructing them here.
 
-**Blocking decision:** auto-proceed — announce without waiting — only when plan depth is **Lightweight AND zero call-outs survive**. Standard and Deep always present the confirmation and wait for it, even with zero call-outs.
+**Blocking decision:** auto-proceed — send the user the auto-proceed announcement in chat, then continue without waiting — only when plan depth is **Lightweight AND zero call-outs survive**. Standard and Deep always present the confirmation and wait for it, even with zero call-outs.
 
 **Headless / opt-in skip:** in headless mode, or when `SKIP_SCOPING_CONFIRM` resolved to skip in Phase 0.0, do not block — compose the internal draft, skip the chat-time confirmation, and write Inferred bets into a `## Assumptions` section when the plan is written (Phase 5.2). The skip covers only this scoping confirmation; Phase 0.4 routing, Phase 0.5 blockers, Phase 2 questions, source-doc disambiguation, and the Phase 5.4 menu still happen. Announcement wording and full routing: `references/synthesis-summary.md` ("Headless mode", "When to skip the blocking confirmation").

skills/ce-plan/references/synthesis-summary.md

@@ -109,21 +109,19 @@ Confirm and I'll proceed to research, drawing on this scope.
 
 The auto-proceed path (announce without waiting for user confirmation) applies only when **plan depth is Lightweight AND there are no call-outs**. For Standard or Deep plans, always ask for confirmation even when there are no call-outs — the plan's substance is what calls for the checkpoint, not how much dialogue preceded it. A Deep plan with rich silent decisions and a 1-3 line summary is exactly the case where rubber-stamping is most likely; the explicit confirmation request gives the user a real chance to push back before research or plan-write proceeds.
 
-When auto-proceed applies (Lightweight + zero call-outs), emit a one-line announcement and continue:
+When auto-proceed applies (Lightweight + zero call-outs), send the user this announcement as a chat message, then continue. Deciding it in your reasoning is not sending it; silent proceeding is not allowed, and the reason (no forks worth flagging) must be visible:
 
 ```
 Planning: [1-3 line summary]
 
 No open decisions to weigh in on — proceeding to [research / plan-write]. Interrupt if I have the scope wrong.
 ```
 
-The announcement is mandatory when skipping — silent proceeding is not allowed. The "why" (no forks worth flagging) must be visible.
-
 For Standard/Deep with no call-outs, the confirmation lines still apply; the "Call outs:" header is simply omitted. The user gets the summary plus the explicit confirmation request.
 
 There is a third skip condition: the **opt-in `SKIP_SCOPING_CONFIRM` setting** (Phase 0.0 — `confirm:auto` token or the `plan_skip_scoping_confirm` config key). When it resolves to skip, the confirmation auto-proceeds for *any* tier or call-out count — the user has pre-authorized it. The announcement is still mandatory (it names that confirmation is off and that inferred scope landed in `## Assumptions`), and the skip is scoped to this confirmation only: genuine blocking questions and the Phase 5.4 menu still run. This differs from headless mode only in that announcement — headless has no synchronous user to announce to.
 
-When the opt-in skip applies, emit this announcement — **not** the auto-proceed template above. The opt-in skip applies to *any* tier and call-out count, so claiming "No open decisions to weigh in on" would be false whenever call-outs survived; the announcement instead names that confirmation is off and that inferred scope is recorded under `## Assumptions`:
+When the opt-in skip applies, send this announcement the same way — **not** the auto-proceed template above. The opt-in skip applies to *any* tier and call-out count, so claiming "No open decisions to weigh in on" would be false whenever call-outs survived; the announcement instead names that confirmation is off and that inferred scope is recorded under `## Assumptions`:
 
 ```
 Planning: [1-3 line scope claim]

tests/skill-eval-cell/fixtures/judged-billing-planned/AGENTS.md

@@ -0,0 +1,2 @@
+# billing-service
+Runs on Kubernetes; cron jobs run unattended at night with no one watching. Plans go in docs/plans/.

tests/skill-eval-cell/fixtures/judged-billing-planned/docs/plans/2026-09-20-0900-feat-nightly-invoice-charging-plan.md

@@ -0,0 +1,41 @@
+---
+title: Nightly charging of overdue invoices
+date: 2026-09-20
+topic: nightly-invoice-charging
+artifact_contract: ce-unified-plan/v1
+product_contract_source: ce-brainstorm
+execution: code
+---
+
+# Nightly charging of overdue invoices
+
+## Goal Capsule
+
+- **Objective:** Overdue open invoices get collected every night without anyone charging them by hand, and no customer is ever charged twice for one invoice.
+- **Product authority:** Billing lead.
+- **Open blockers:** None.
+
+## Product Contract
+
+### Summary
+
+An unattended nightly job charges each overdue open invoice to the customer's saved card and marks it paid only when the charge succeeds. The billing team gets a morning list of failed charges with reasons.
+
+### Problem Frame
+
+Every morning someone on the billing team spends about an hour charging overdue invoices by hand in the payment provider's dashboard. The work is mechanical, and when it is skipped, collection slips.
+
+### Requirements
+
+- R1. Each night, charge every overdue open invoice to its customer's saved card.
+- R2. Mark an invoice paid only when its charge succeeds.
+- R3. A customer is never charged twice for the same invoice, including when a run is retried or overlaps another run.
+- R4. Each morning the billing team can see which charges failed and why.
+
+### Scope Boundaries
+
+- Automatic retry schedules, customer emails, an admin UI, and audit logs are out of scope for this release.
+
+### Key Decisions
+
+- Failed charges stay open for the team to follow up by hand, rather than being retried automatically.

tests/skill-eval-cell/fixtures/judged-billing-planned/src/invoices.ts

@@ -0,0 +1,3 @@
+// invoices: id, customer_id, amount_cents, due_at, status ('open' | 'paid' | 'void').
+export async function findOverdueOpenInvoices(): Promise<{ id: string; customerId: string; amountCents: number }[]> { return [] }
+export async function markPaid(invoiceId: string): Promise<void> {}

tests/skill-eval-cell/fixtures/judged-billing-planned/src/payments.ts

@@ -0,0 +1,2 @@
+// Stripe wrapper. chargeSavedCard creates a PaymentIntent and confirms it off-session.
+export async function chargeSavedCard(customerId: string, amountCents: number, opts: { idempotencyKey?: string } = {}) { /* ... */ }

tests/skill-eval-cell/judged/personas/job-planned.md

@@ -0,0 +1,7 @@
+You lead billing at a small SaaS company.
+Opening request: plan the nightly invoice-charging work from the brainstorm doc in docs/plans/.
+Why: every morning someone spends an hour charging overdue invoices by hand in the payment provider's dashboard. You want that hour back.
+What you already decided in the brainstorm: charge the saved card, mark paid only on success, never charge twice, a morning list of failures with reasons, no retries, emails, admin UI or audit logs.
+Needs you will not volunteer but will confirm if the assistant raises them: the morning list can just be a log line or a report the job writes; nobody wants a new page. Invoices voided during the day must not be charged that night.
+What you would call overkill if proposed: a job-queue framework, a new database table for runs, a dashboard, alerting integrations.
+Style: short answers. For technical choices you say it's the assistant's call.

tests/skill-eval-cell/judged/rubrics/scoping-confirmation.md

@@ -0,0 +1,21 @@
+# Grading rubric: the scoping confirmation
+
+You grade one planning or brainstorming conversation. You do not know how it was produced. You get the user's persona (the ground truth: their goal, the reason behind it, needs they will not volunteer, and what they would call overkill), the full conversation, and any document the assistant wrote.
+
+Find the scoping confirmation: the message, sent before the document is written or updated, where the assistant states the scope it will write against and either asks the user to confirm it or announces that it is proceeding without waiting. Its timing depends on the run: a plan made without a brainstorm doc usually confirms before research, while a plan built from a brainstorm doc confirms after research, just before the write. A status update or a list of findings that asks for no confirmation is not it. If the assistant showed more than one (after a revision), grade the first. If there is none, set "shown" to false and grade the rest as null or 0.
+
+1. shown: whether a scoping confirmation appeared before the document was written.
+2. problem_stated: whether the confirmation states the problem or goal behind the request (who has it, what it costs, or what will be true when done), as opposed to only describing the feature or scope. A feature description with a "so that" clause counts only if the clause names the real goal.
+3. problem_correct: when problem_stated is true, whether it matches the persona's reason without distortion; null otherwise.
+4. redirectable: count the items in the confirmation that are real calls the user could confirm or change (a fork in approach, a non-obvious inclusion or exclusion, a bet the assistant made without asking, a consequence of combining the user's answers).
+5. noise: count items that carry nothing the user could act on: restating what the user just said, mechanical choices with no real alternative, completeness counts, or process narration.
+6. needs_code: count items the user could not judge without reading code: file paths, function or table names, data shapes, line numbers, exact error wording, bare requirement IDs such as R3 or AE2.
+7. bullets: total bullet points in the confirmation.
+8. words: approximate word count of the confirmation message.
+9. one_read: 1 to 5. Could this persona, reading once, tell whether the assistant understood them and what to push back on? 5 means immediately; 1 means they would have to reread or would likely rubber-stamp.
+10. corrected: whether the user's reply to the confirmation corrected or redirected anything.
+11. hidden_in_confirmation: count of the persona's "will not volunteer" needs that the confirmation itself reflects.
+
+Return only one JSON object, with no code fence or other text:
+
+{"metrics": {"shown": true, "problem_stated": true, "problem_correct": null, "redirectable": 0, "noise": 0, "needs_code": 0, "bullets": 0, "words": 0, "one_read": 0, "corrected": false, "hidden_in_confirmation": 0}, "items": [{"kind": "noise|needs_code|redirectable|problem", "text": "one short line"}], "widened": "the most important thing the confirmation surfaced that the user had not raised", "pushback": "what the user corrected, or none"}