feat(ce-polish): live mode (EveryInc#1726)

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Code <noreply@anthropic.com>

Author: 3 people

docs/guides/ce-polish.md

@@ -4,6 +4,8 @@
 
 `ce-polish` is live UX polish for a feature that already works. It starts the project's dev server, opens the verified URL through the active harness when it can, and then waits. You use the page and name what is off. It edits, hot reload updates the page, and you keep going until you say you are done. Then it commits on the current branch and stops. No PR.
 
+It asks one question first: **traditional** or **live**. Traditional is the typed loop above. Live puts a voice interviewer and a drawing layer into the page through [riffrec](https://github.com/kieranklaassen/riffrec): you talk and draw, units appear on a board as you speak, and the agent acts on them in batches at checkpoints. See [Live mode](#live-mode).
+
 It is not `ce-prototype` (decide how something should feel before it exists), not `ce-simplify-code` (trim recently changed code), and not `ce-dogfood` or `ce-test-browser` (autonomous QA or a test pass). Polish is a conversation with a running app.
 
 Manual invocation only. It starts a server and runs the checked-out branch, so it waits for you to type it.
@@ -64,6 +66,28 @@ There is no scoring rubric and no checklist. You name what is wrong; it changes
 
 ---
 
+## Live mode
+
+Live mode is the same loop with your voice and your cursor as the input. It needs three things: `OPENAI_API_KEY` in the environment (the interviewer runs on OpenAI Realtime, from the browser, on a short-lived secret the skill's local endpoint mints), a React app (Vite, Next, Remix, or Rails with Inertia React), and riffrec in that app. When riffrec is missing, the skill says so up front and adds it as one setup commit on your current branch: the dependency (installed from riffrec's GitHub repository, pinned to a known-good commit, until riffrec publishes a release with live mode; the skill then switches to the npm range in a later setup commit) and a `<RiffrecProvider forceEnable live={{}}>` mount at the app root. The consent step opens only on a page opened from the session URL; ordinary visits see nothing. That commit stays after the session; riffrec is meant to remain in the app.
+
+**What runs.** A small Node endpoint bundled with the skill receives the session stream, mints the interviewer's secret, and wakes the agent at checkpoints. The dev server runs as in traditional mode. You open the app URL with a fragment carrying a per-session token and the endpoint origin; the page strips it before it can land in history. A consent screen names what goes where: microphone audio, a short session brief (routes, component names, design tokens, recent changes; never file contents or secrets), what you click, draw on, and pin, and a screenshot of the page whenever you point at something or ask the interviewer to look go to OpenAI; transcript, screenshots, frames, and events go to the endpoint on your machine. Decline, and nothing starts; the skill offers traditional and names the setup commit that remains.
+
+**The loop.** Speak a change while pointing at or drawing on the element. The interviewer extracts a unit, asks right away when the instruction is ambiguous, and shows it on the board. Units are released to the agent as a batch when you pause, change pages, or press Send. The agent acknowledges the batch, then acts by the mode switch on the board:
+
+| Mode | Clear bounded edit | Ambiguous unit | Beyond polish |
+|------|--------------------|----------------|---------------|
+| **Instant** | applied now | best reading applied, guess noted on the unit | residual list |
+| **Smart** (default) | applied now | question voiced by the interviewer; unit waits | residual list |
+| **Collect** | held | question voiced; held | residual list |
+
+A mode change takes effect at the next checkpoint; moving the switch off Collect releases the held, accepted units to the agent right away under the new mode. Questions come back through the interviewer at your next pause, and your spoken answer resumes the unit. If an edit breaks the page and it does not come back, the agent fixes or reverts before it waits again. Nothing is acted on before a checkpoint, however fast it appears on the board.
+
+**Session end.** Press Done on the board; it asks you to confirm each unit's intended element and change, then sends the final batch. Collect applies its held batch in one pass. The agent commits through `ce-commit`, writes a residual list (units that were blocked, still waiting on an answer, or beyond polish, with their anchors) next to the session, and reports the commits, the app URL, the residual path, and the session log path. The log holds the full session with your per-unit confirmations and can be replayed later under a different evidence profile.
+
+**Remote sessions.** The browser can be on another machine. Direct LAN or tailnet URLs over plain HTTP work for drawing and the board, but browsers refuse the microphone and screen capture on an insecure page, so voice needs HTTPS for both the app and the endpoint, which means two tunnels (Tailscale serve, cloudflared, or ngrok). The skill refuses to pair an HTTPS page with a plain-HTTP endpoint and tells you which origin has to change.
+
+---
+
 ## Quick Example
 
 The notification settings page works. Spacing is tight, the off toggle is easy to miss, and the empty-state copy is dry. You run `/ce-polish` on the feature branch.
@@ -116,6 +140,8 @@ Nothing in the core loop calls this. `ce-explain` may suggest it; you still type
 
 Required: a startable local dev server. Browser opening and inspection come from the active harness; with neither, it prints the URL and you describe what you see. Every form stops on a requested branch that cannot be reached without moving user changes or creating a worktree behind the harness.
 
+Live mode additionally requires `OPENAI_API_KEY` in the environment, Node, and a React app. Without the key it names the variable and offers traditional; on a non-React project it says live mode needs a React app.
+
 ---
 
 ## FAQ
@@ -132,6 +158,12 @@ Same answer. Framework detection and server start do not depend on the browser h
 **Why no PR at the end?**
 Polish is often more than one sitting. Commit-and-PR is `/ce-commit-push-pr`.
 
+**Does live mode send my code to OpenAI?**
+No source code. OpenAI receives microphone audio, the session brief (route names, component names, design token names, a paragraph on recent changes), what you click, draw on, and pin, and a screenshot of the page when you point at something or ask the interviewer to look, so whatever your app shows on screen at that moment goes with it. The continuous frames, transcript, and events go only to the endpoint running on your machine. The consent screen lists exactly this before anything starts.
+
+**What if I decline the consent screen?**
+No session starts. The endpoint stops, the skill offers traditional polish, and it names the riffrec setup commit if it made one. That commit is not reverted.
+
 ---
 
 ## See Also
@@ -143,3 +175,4 @@ Polish is often more than one sitting. Commit-and-PR is `/ce-commit-push-pr`.
 - [`ce-dogfood`](./ce-dogfood.md): autonomous browser QA of the branch, with fixes
 - [`ce-commit-push-pr`](./ce-commit-push-pr.md): open the PR after polish
 - [`ce-debug`](./ce-debug.md): a bug you find during polish that needs a causal chain
+- [`ce-riffrec-feedback-analysis`](./ce-riffrec-feedback-analysis.md): analyze a recorded riffrec zip after the fact instead of live

skills/ce-polish/SKILL.md

@@ -7,15 +7,16 @@ argument-hint: "[PR number, branch name, or blank for current branch]"
 
 # Polish
 
-Put a working feature in front of the user and turn their live observations into focused UX fixes on the running page.
+Put a working feature in front of the user and turn their live observations into focused UX fixes on the running page. Two ways to collect those observations: **traditional**, where the user types what could be better; and **live**, where they talk and draw on the page while a voice interviewer turns speech into units and you act on them at checkpoints.
 
-**Done:** the user ends the polish loop, every requested fix is reflected in the live feature or reported as blocked, and the in-scope changes are saved in local commit(s). A server or checkout blocker also ends the run when it is reported with the evidence needed to resume.
+**Done:** the user ends the polish loop, every requested fix is reflected in the live feature or reported as blocked, and the in-scope changes are saved in local commit(s). A live session also ends with a residual list of the units that were not applied and the path of its session log. A server or checkout blocker also ends the run when it is reported with the evidence needed to resume.
 
-**Boundaries:** the user drives what to inspect and change; do not invent an autonomous checklist or expand into general QA. Never work on the repository's default branch. This workflow may edit and locally commit the requested polish, but it never pushes or opens a PR.
+**Boundaries:** the user drives what to inspect and change; do not invent an autonomous checklist or expand into general QA. Never work on the repository's default branch. This workflow may edit and locally commit the requested polish, and in live mode the disclosed riffrec setup commit, but it never pushes or opens a PR.
 
 ## Run
 
-1. **Get the live page ready.** Read `references/run.md` before resolving the requested ref or starting anything. It owns existing-worktree safety, dev-server discovery, the bundled-script calls, reachability, and the browser handoff.
-2. **Wait for observations.** Tell the user where the server is running and ask what could be better. Do not start a review pass while they browse.
-3. **Iterate.** For each requested change, inspect only as needed, edit the in-scope surface, and let hot reload update the page. When the user asks you to inspect the result, use a browser capability available in the active harness; if none exists, ask them to describe what they see.
-4. **Close locally.** When the user says they are done, invoke `ce-commit` for the polish changes, then report the commit(s), the still-running server URL, and any residual blocker.
+1. **Ask live or traditional.** Ask once, before any server starts. Include this disclosure with the question: live mode needs an OpenAI key in the environment, runs a voice interviewer in the page that hears the user, is told what they click and draw on, and looks at their screen when they point at something or ask it to, streams the session to a local endpoint this skill runs, and, when riffrec is not yet in the app, adds the riffrec dependency and a provider mount as a setup commit on the current branch that stays after the session. Traditional continues with steps 2–5 unchanged. Live: read `references/live-start.md`; it owns preconditions, install, the endpoint, and the URL handoff, applies the workspace and server rules of `references/run.md` itself, then routes to `references/live-loop.md` for the session and its close. Steps 2–5 do not run for a live session.
+2. **Get the live page ready.** Read `references/run.md` before resolving the requested ref or starting anything. It owns existing-worktree safety, dev-server discovery, the bundled-script calls, reachability, and the browser handoff.
+3. **Wait for observations.** Tell the user where the server is running and ask what could be better. Do not start a review pass while they browse.
+4. **Iterate.** For each requested change, inspect only as needed, edit the in-scope surface, and let hot reload update the page. When the user asks you to inspect the result, use a browser capability available in the active harness; if none exists, ask them to describe what they see.
+5. **Close locally.** When the user says they are done, invoke `ce-commit` for the polish changes, then report the commit(s), the still-running server URL, and any residual blocker.

skills/ce-polish/references/install-riffrec.md

@@ -0,0 +1,37 @@
+# Install riffrec live mode into the host app
+
+Load this from `references/live-start.md` when the detect script shows the app is missing the riffrec dependency, has no installed live-capable build, or has no live-capable provider mount. The result is one setup commit on the current feature branch that leaves the app able to run a live session; the riffer already accepted this when they chose live mode, so do not ask again.
+
+## Where riffrec comes from
+
+`RIFFREC_MIN_COMMIT` = `1393c17082e3b6a5b04d219fc6ae9083018c96a2`
+`RIFFREC_MIN_VERSION` = unset (no npm release carries live mode yet)
+
+Live mode with the seeing interviewer (clicks announced, `look_at_screen`, no zip once the endpoint confirmed Done) is complete on riffrec's GitHub `main` from `RIFFREC_MIN_COMMIT` on, and the tree ships its built `dist/`, so a git install pinned to that commit is the install path (step 1). The registry is the future switch: when riffrec publishes a release that carries live mode, that release becomes `RIFFREC_MIN_VERSION` and a new setup commit replaces the git pin with the registry range `^RIFFREC_MIN_VERSION`. Until then nothing here reads the registry, and `RIFFREC_MIN_VERSION` being unset neither blocks nor gates the install. Name the pinned commit in the setup commit message and in the report.
+
+## What to change
+
+Three edits, together, in the project root the detect script inspected:
+
+1. **Dependency.** Use the package manager the detect output named (`package_manager`; when null the project has no `package.json` and live mode cannot host riffrec — report that and offer traditional). Let the manager update its lockfile; do not edit the lockfile by hand.
+   - **Git install, pinned (the path today):** one command, the manager's own add verb followed by the git spec with the full SHA: `<add verb> kieranklaassen/riffrec#1393c17082e3b6a5b04d219fc6ae9083018c96a2`, where the add verb is `npm install`, `pnpm add`, `yarn add`, or `bun add` for the manager the detect output named. Write the SHA out in full; a shortened ref is a different, nonexistent revision to the manager. An existing `riffrec` entry that points anywhere else (a registry range, an older commit, `#main`) is replaced by the pin. The repository declares only a `prepack` build, which git installs never run, and ships its built `dist/` in the tree, so what you get is exactly the committed `dist/`.
+   - **Check the build.** Re-run the detect script: `installed` and `live_build` must both be true. `live_build` requires the installed `dist/index.d.ts` to name both `RiffrecLiveConfig` (the live provider) and `LOOK_AT_SCREEN_TOOL` (the page-side handler for the screen tool the endpoint advertises to the interviewer); an older live-capable build has the first without the second and would leave the interviewer's screen requests unanswered, so it fails this check and is replaced by the pin like any other stale entry. If `live_build` is false the install did not resolve to the pin (a lockfile still holding an older resolution, a manager cache); fix the resolution and reinstall rather than building inside `node_modules`. Still false after that: stop, report the command output, and offer traditional. The detect script's `version` is informational (it reads the installed `package.json`, not a registry); the probe in `references/live-start.md` (the consent screen appears) is what proves live mode is present.
+   - **Future switch (not this session):** when riffrec publishes a release that carries live mode, `RIFFREC_MIN_VERSION` names it and a new setup commit swaps the git pin for `riffrec@^<min>` with the same add verb.
+2. **Mount.** Wrap the root of the React tree once in the provider, imported from `riffrec`:
+
+   ```tsx
+   <RiffrecProvider forceEnable live={{}}>
+   ```
+
+   `live` enables live mode. Its `autoStart` default is fragment-gated in riffrec from `RIFFREC_MIN_COMMIT` on: the consent step opens by itself only when the page carries live credentials (the `#riffrec_live=…` fragment on the handoff URL, or stored credentials after a reload), and an ordinary visit stays quiet, so leave `autoStart` unset. The root is wherever the framework renders the whole app: the Vite entry that calls `createRoot`, the Remix `root.tsx` layout, the component Inertia's `createInertiaApp` renders, or, on the Next app router, a client component the root layout renders (the provider uses browser APIs, so it cannot sit directly in a server component). When a `RiffrecProvider` mount already exists, add the `live` prop to it and keep its other props. That prop is the whole configuration: the endpoint origin and the session token reach the page through the URL fragment at session start, so nothing about the endpoint is written into source, environment files, or config, and the mount is safe to commit.
+3. **Verify.** Re-run the detect script from `references/live-start.md`. It must now report `dependency`, `installed`, and `live_build` true, and `mount` true (`live=` on the tag; the detect script looks for a real JSX opening element, not the prop's value). If the mount edit does not show, the file you edited is not the rendered root; find the one that is.
+
+Follow riffrec's own README when it names a different mount point for the framework in front of you; the README is the source of truth for the package, this file only for what polish needs from it.
+
+## Commit
+
+Commit the dependency change, the lockfile, and the mount edit as one setup commit on the current branch through `ce-commit`, with a message that says it adds riffrec live mode for polish sessions. Keep its hash: the live-mode report names it, including when the riffer later declines the consent screen and no session runs (the commit stays; riffrec remains in the app after the session, as disclosed).
+
+## When it fails
+
+An install or mount edit that cannot complete leaves no setup commit. Undo only the edits this step made, report what failed with the command output, and offer traditional polish. Do not retry with a different package manager or a different riffrec source than the pinned commit.