diff --git a/.flatbread-proof/findings/fnd-a-stale-decision-costs-builds-more-than-proof-gi--pdrkn9kjkejk52kk.md b/.flatbread-proof/findings/fnd-a-stale-decision-costs-builds-more-than-proof-gi--pdrkn9kjkejk52kk.md new file mode 100644 index 00000000..1e64fe27 --- /dev/null +++ b/.flatbread-proof/findings/fnd-a-stale-decision-costs-builds-more-than-proof-gi--pdrkn9kjkejk52kk.md @@ -0,0 +1,19 @@ +--- +id: fnd-a-stale-decision-costs-builds-more-than-proof-gi--pdrkn9kjkejk52kk +effort: eff-effort-graph-memory-and-agent-wedge--szeqvmgqjqnhd002 +title: A stale Decision costs builds more than Proof gives them +kind: measurement +created_at: '2026-10-21T07:20:17.954Z' +derives_from: + - fnd-stale-code-anchors-cost-little-and-rename-aware--2kyseq5ryvsxc9jc +--- + +A feature-building benchmark (benchmarks/proof-build) asked Haiku to build or plan features in a small TypeScript editor from product-sense requests, with the same code and request in every arm and only Proof access changing: none, the skill alone, find, find with --path and --refs, every brief pasted, and every live record pasted. 18 build tasks ran 3 times and 8 plan tasks twice per arm (446 sessions). Hidden acceptance, constraint and regression tests graded builds; a blinded judge scored plans. + +- Builds all-green: none 47 of 54 (87%), skill 41, find 40, findcode 41, brief 37, dump 36 (67%). Processed tokens rose to 1.3x, 1.5x and 2.0x of none as more Proof was forced in. +- One stale accepted Decision, saying shortcuts are registered in a file a refactor had turned into the frozen plugin table, caused most of the loss: every task that adds a shortcut broke plugin API v1 in Proof arms (B-S3: none 3 of 3, every Proof arm 0 of 3). findcode sessions saw the stale-anchor note and followed the Decision anyway. +- Proof-only build tasks showed no gain (none 17 of 18): the code's own patterns taught the recorded rules. +- Plans: pasted Proof lifted proof-only plans (dump 1.00 against 0.90) and dragged stale ones (0.76 against 1.00); overall equal. +- Not significant on the pre-registered tests (best Holm p 0.31; pooled dump p 0.059); counted per task, 1 against 4 tasks, because the harm sits in tasks sharing one record. + +Proof in context did not improve building here, and an out-of-date Decision cost more than flagging its anchor could recover. A Decision whose evidence or code has moved needs to read as suspect itself. diff --git a/benchmarks/proof-build/DESIGN.md b/benchmarks/proof-build/DESIGN.md new file mode 100644 index 00000000..c97137b5 --- /dev/null +++ b/benchmarks/proof-build/DESIGN.md @@ -0,0 +1,598 @@ +# Proof build benchmark: pre-registered design + +Status: pre-registered on 2026-10-20, before any bed, task or paid session +exists. Changes after the pilot are allowed only where this file says so, +and each one is logged at the end with its reason. + +## Question + +When an agent builds a feature from a product-level request, does putting +Proof records in its context make the work better or worse? Proof can help +when the right solution keeps a rule that only Proof records. It can hurt +when Proof is out of date, or when reading it costs attention that the code +needed. We compare six arms (no Proof, Proof on request, Proof forced in, +search before planning, search plus code paths, everything pasted in) on +the same code and the same requests, and score the results with hidden +tests (build tasks) and a blind rubric (plan tasks). + +Pre-registered hypotheses: + +- **H1 (help).** On proof-only tasks, `skill`, `find` and `findcode` are + all-green (acceptance, constraints and regressions all passing) more + often than `none`. A constraint pass counts only in a session that also + passed acceptance, since constraint tests skip when the feature is + missing. +- **H2 (redundant).** On code-visible tasks, no arm differs from `none` on + all-green. Proof only adds cost there. +- **H3 (harm).** On stale tasks, `brief` and `dump` do worse than `none` on + all-green. `skill`, `find` and `findcode` do no worse than `none`. +- **H4 (burden).** On control tasks, all-green is the same in every arm. + Context tokens, processed tokens and wall time rise with the amount of + Proof forced in (`none` ≤ `skill` < `find` ≤ `findcode` < `brief` < + `dump` < `dump10x`). These measures include the first turn, which holds + the pasted brief or dump; read tokens leave it out and are reported as + the session's own reading. + +## 1. The bed app: Tessera core + +Tessera is a fictional design editor. Its core is zero-dependency +TypeScript with no DOM, run by Node 22 with `--experimental-strip-types`. +It uses erasable syntax only: no enums, namespaces or parameter +properties, and imports use explicit `.ts` extensions. It has about 68 +source files and roughly 6,000 lines, plus about 16 test files and 1,600 +lines of `node:test` tests (`node --experimental-strip-types --test`, +which picks up `test/**/*.ts`; passing `test/` fails on Node 22.22). There +is no `package.json` dependency and no build step. + +`src/editor.ts#createEditor()` is the stable facade that every hidden test +drives: `doc`, `selection`, `history`, `panel` (`view()`, `edit(label, text)`), `layers`, `keys` (`press('Mod+Shift+H')`, `focus(scope)`), +`palette` (`run(title)`, `list()`), `plugins` (`load(manifest)`), and +`files` (`save()`, `load(json)`, `viewState()`). Hidden tests use only this +facade, palette titles, field labels, shortcut keys and plugin event names, +which are the names a user or plugin author sees. Because of that, any +reasonable implementation can pass. + +| Module | Holds | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `src/util/` (`ids`, `emitter`, `assert`, `math`) | id minting, a typed emitter, small math | +| `src/model/` (`types`, `document`, `tree`, `factory`, `geometry`, `transform`, `style`, `text`, `query`) | frames, rects, ellipses, lines, text and groups; parent/child operations; bounds; rotation (stored in radians since the drift commit, see §2); fills, strokes and opacity, with groups owning their own opacity since the drift commit; text runs | +| `src/units/` (`registry`, `expr`, `parse`, `precision`, `format`) | the unit registry (px, pt, in, %); the expression parser the panel uses (`expr.ts`, which handles arithmetic); `parse.ts`, a legacy wrapper now used only to read old string values in files; display precision per unit | +| `src/selection/` (`selection`, `hit`, `rules`, `bounds`) | the selection set; point hit-testing that skips hidden and locked nodes; selection bounds | +| `src/commands/` (`types`, `bus`, `history`, `coalesce`, `batch`, `registry`, `palette`) and `builtin/` (`move`, `resize`, `set-prop`, `reorder`, `group`, `delete`, `rename`, `visibility`, `lock`, `nudge`) | the command bus; undo/redo with a selection before and after each entry; gesture coalescing by gesture id; `batch()`, used only by `group`; palette titles | +| `src/panel/` (`properties`, `mixed`, `fields`, `commit`, `schema`) and `sections/` (`layout`, `fill`, `stroke`, `text`, `appearance`) | the properties-panel view model: sections, field descriptors, the `Mixed` sentinel, commit per field | +| `src/layers/` (`layers`, `tree-view`, `drag`, `rename`) | the layers-panel view model, listed frontmost first since the drift commit; expand and collapse; drag to reorder | +| `src/keys/` (`keymap`, `dispatcher`, `scopes`, `chords`) and `bindings/` (`canvas`, `layers`, `panel`) | the live bindings for each scope (`bindings/`, since the split); `keymap.ts` keeps `DEFAULT_KEYMAP`, a frozen legacy snapshot exported to plugins; focus scopes | +| `src/persist/` (`schema`, `serialize`, `deserialize`, `viewstate`) and `migrations/` (`index`, `v1-v2` … `v4-v5`) | an integer schema version (5); migrations applied one step at a time; a view-state sidecar (zoom, collapsed sections) kept out of the file | +| `src/plugin/` (`api`, `host`, `events`, `capabilities`, `sandbox`) | the frozen v1 plugin API (a header comment says frozen, and a surface snapshot test checks it); events that carry ids only; one capability (`export`) | +| `docs/plugins.md`, `AGENTS.md` | the plugin reference (names only, no rationale); how to run the tests | + +## 2. History and the Proof graph + +`scripts/gen-bed.mjs` builds the bed deterministically from +`spec/history.mjs` (ordered commits, each one a set of whole-file writes +with a fixed author, date and message) and `spec/proof.mjs` (the records, +as mutations replayed through the real `createProofWriter`, the way +`proof-recall/scripts/build-repo.mjs` does it). + +- **Commits.** 84 commits, from 2026-02-02 to 2026-06-12, by four authors: + Ines Okafor (model, persistence), Tomas Lindqvist (commands, history), + Priya Raman (panel, units) and Mateo Alves (layers, keys, plugins). A + fifth author, Ren Ito, joins in May and makes the drift commits. About 36 + commits implement a record and carry `Proof: ` trailers. The others + are fixes, tests and chores with ordinary subjects. Commit messages never + state a rationale that only Proof holds. +- **Variants.** `--variant proof` writes `.flatbread-proof/`, + `flatbread.config.js` and trailers. `--variant none` writes the same + commits with no trailers and no Proof files. The two variants have + identical code trees, and `check.mjs` verifies that with `git ls-tree`. + Hashes differ between variants. Citations pin hashes from the proof + variant. +- **Efforts.** There are seven, one for each area: Document model and + geometry; Units and precision; Command bus and history; Properties panel; + Selection, layers and shortcuts; Persistence and migrations; Plugin API + v1. Each has about 20 live records (around 5 Decisions with rationale + and rejected alternatives, 2 Constraints, 3 Findings, 2 Issues mostly + resolved, 1 or 2 Risks, 4 to 6 Citations `path#symbol@sha`, and one + compaction) plus 3 or 4 dead ones (superseded, rejected, invalidated, + retracted). That is about 140 live and 25 dead records. Two compactions + are stale (`compaction.stale`). +- **Proper supersession (Proof is right).** Coalescing by a 500 ms time + window was superseded by coalescing by gesture id, with the rationale + that keyboard nudges and slow drags merged wrongly. The undo cap went + from 100 to 200. +- **Drift (Proof is wrong, code is the truth).** Five late commits with no + trailer, none of which updates Proof: + + | Key | Kind | Proof still says | Code now | + | --- | ----------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | + | D1 | unrecorded supersession | "Groups carry no style; opacity on a group applies to its children" | groups have their own `opacity`, multiplied at render; tests updated | + | D2 | unrecorded supersession | "Rotation is stored in degrees" | radians; the panel converts | + | D3 | unrecorded supersession | "The layers list is back-to-front, index 0 is the backmost layer" | frontmost first; drag indexes reversed | + | D4 | refactor drift | Citations and a Decision name `keys/keymap.ts#DEFAULT_KEYMAP` as where shortcuts are registered | bindings live in `keys/bindings/*`; `DEFAULT_KEYMAP` is a frozen snapshot the dispatcher never reads | + | D5 | refactor drift | Citations and a Decision name `units/parse.ts#parseLength` as the parser for panel input | the panel uses `units/expr.ts#parseExpression`; `parseLength` only reads legacy file strings | + +- **Rules planted in Proof.** Each rule is a Decision or Constraint with + its rationale. Most also have an Issue or Finding that motivated them + and a Citation. + + | Key | Rule | Visible in code or tests? | + | --- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | + | R1 | one user gesture is one history entry, however many nodes it touches (Issue: "undo needed 12 presses after a multi-select edit") | no (`batch()` exists, unexplained, and only `group` uses it) | + | R2 | command payloads are plain JSON holding node ids, never node objects (macro recording, replay) | in effect (the `Json` payload type, `commands.replay`, `groupNodes`) | + | R3 | a shipped migration is never edited; a schema change adds a new step (Issue: files diverged after an edited step) | no | + | R4 | coalesce by gesture id, never by a time window (the head of the supersession above) | in effect (the gesture-id API, `KeyEvent.repeat`, a test naming it) | + | R5 | plugins receive frozen snapshots, never live nodes (Issue: a plugin mutated nodes and bypassed history) | partly (event payloads are frozen at the top level; copies, deep) | + | R6 | view state (expanded groups, zoom) never goes into the document file; it goes in the sidecar | no (the sidecar exists, with no comment) | + | V1 | hidden and locked nodes are never hit or selected from the canvas | yes (the filter in `hit.ts` and a test) | + | V2 | stored geometry is never rounded; precision applies only to display | yes (a comment in `precision.ts` and a test that ten 0.1 nudges equal 1) | + | V3 | undo restores the exact selection from before the change | yes (the entry shape in `history.ts` and a test for `group`) | + | V4 | plugin API v1 is frozen | yes (a header comment and a surface snapshot test) | + +- **Filler at 10x** (`--scale 10`, used only by `dump10x`). Nine filler + Efforts in look-alike areas (export, prototyping, comments, assets, + colour management, collaboration cursors, onboarding, telemetry, fonts) + bring the graph to about 1,500 records. They are written in the same + voice and cite only filler paths that do not exist in the bed. They never + state a rule about the seven real areas. The code is identical. +- **No anticipation.** Records describe past work and standing rules. An + offline check fails the build if any record body, title or commit message + contains a task's feature phrases (for example "corner radius" or + "marquee"). Area words such as "properties panel" or "undo" are allowed. + +## 3. Tasks (26: 18 build, 8 plan) + +Prompts describe intended behaviour in a user's words. They name no file, +symbol, module, Effort or record id. They may name product surface: palette +titles, panel field labels, shortcut keys, and plugin event names as +`docs/plugins.md` lists them. Each task is +`p4/tasks//{task.json, acceptance.test.ts, constraints.test.ts, reference.patch, naive.patch}`, outside the repository. `task.json` holds +the id, kind, category, prompt, rule keys, `leakWords` (phrases that must +not appear in the none variant) and `featureWords` (phrases that must not +appear in Proof). + +### Build tasks (18) + +The list as it stands after the review before any run; the change log +says what changed from the pre-registered list and why. + +| Id | Category | Request (gist; the real prompt is 3 to 6 sentences of behaviour) | Rule | Naive patch | +| ---- | ------------ | -------------------------------------------------------------------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------ | +| B-P1 | proof-only | Rename layers: a base name numbers the selected layers in selection order; `#` marks where the number goes; undoable | R1 | one rename command per layer, so one Undo per layer | +| B-P3 | proof-only | A Layout grid field on frames (Off or a size), saved with the file; files of any older version open with the grid off | R3 | a load-time default for the new field, with no new step or version | +| B-P5 | proof-only | A `selectionchange` plugin event with `{ layers }` (id, name, type, x, y, width, height) in selection order | R5 | plain copies in a payload frozen only at the top, like the existing emitters | +| B-P6 | proof-only | After reopening a document, the layers panel shows the same groups and frames expanded | R6 | an `expanded` flag on group and frame nodes, saved in the document file | +| B-P7 | proof-only | Copy style and Paste style: fills, strokes and opacity from one layer onto the selection; groups left alone; undoable | R1 | one set-prop command per property and layer, so a paste takes many Undos | +| B-P8 | proof-only | Old files that name colours (red, navy, …) open with hex colours, including files saved again since | R3 | the names mapped inside the shipped v2→v3 step, plus a load-time fix, with no new version | +| B-V1 | code-visible | Dragging on empty canvas draws a selection box; covered groups are taken whole, partly covered ones give up their covered layers | V1 | the walk skips the hit filter, so hidden and locked layers are selected | +| B-V2 | code-visible | Scale selection by a typed percentage about the selection's centre, contents included | V2 | stored values rounded to 2 decimals | +| B-V3 | code-visible | Ungroup (palette and Mod+Shift+G): children take the group's place and become selected; one Undo restores the groups | V3 | the freed layers are selected before the batch, so Undo does not reselect the groups | +| B-V4 | code-visible | Duplicate (palette and Mod+D): copies 10 px right and down, in front of the originals, selected; Undo and Redo | R2, V3 | the copies are selected after the batch, so Redo brings them back unselected | +| B-V5 | code-visible | Holding an arrow key keeps nudging; one Undo takes back the whole hold; letting go ends the step | R4 | repeats within 400 ms by the editor clock merge, so a hold whose repeats pause splits | +| B-S1 | stale (D1) | Number keys 1–9 and 0 set the selected layers' opacity | D1 | follows Proof: a group's value is written to the layers inside it | +| B-S2 | stale (D2) | Shift while rotating snaps to 15° steps; the Rotation field shows the snapped angle | D2 | follows Proof: the snap is done and stored in degrees | +| B-S3 | stale (D4) | Mod+Shift+H hides or shows the selected layers; the step is labelled Hide or Show | D4 | follows Proof: the chord is also added to `DEFAULT_KEYMAP`, changing the frozen v1 table | +| B-S4 | stale (D5) | W and H accept in, pt, px and % of the enclosing frame, mixed with arithmetic | D5 | follows Proof: `parseLength` is extended and tried first, so a unit inside arithmetic is refused | +| B-K1 | control | Read-only Characters and Words fields in the Text section | none | the fields in a new section, which breaks an existing test | +| B-K2 | control | Fill and Stroke accept `#f80`, six-digit hex with or without `#`, and `rgb(r, g, b)`; shown as upper-case `#RRGGBB` | none | `parseColor` itself widened, which breaks an existing test | +| B-K3 | control | Distribute horizontally or vertically with an optional typed gap | none | a missing gap read as 0, which breaks an existing test | + +### Plan tasks (8) + +The agent writes `PLAN.md` and changes no code. Each plan task needs at +least two rules and is scored on a rubric (§5). + +| Id | Category | Request (gist) | What it hinges on | +| ---- | ------------ | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| L-P1 | proof-only | Components: turn a frame into a reusable main; instances update; text and fill overrides | R1 (an override edit is one entry), R3 (a new migration step), R5 (plugins get snapshots of instances), V4 (no v1 fields) | +| L-P2 | proof-only | Version history: browse and restore earlier versions of a document | R2 (replayable commands), R6 (view state excluded), R3 | +| L-P3 | proof-only | Auto layout: frames stack children with gap and padding | R1 (relayout belongs to the edit's entry), R4, V2 | +| L-V1 | code-visible | Snapping to other layers' edges and centres while dragging | V1 (no snapping to hidden or locked layers), V2 (snap shown, not rounded in storage) | +| L-S1 | stale | Masks: a shape clips the layers above it in its group | D3 (frontmost-first order), D1 (groups have their own opacity); must not plan on back-to-front indexes | +| L-S2 | stale | Users can rebind shortcuts in preferences | D4 (rebinding goes through `bindings/` and the dispatcher; `DEFAULT_KEYMAP` stays frozen) | +| L-K1 | control | Rulers along the top and left with nicely spaced labelled ticks for the current zoom | none | +| L-K2 | control | A document swatches list built from the fills in use, sorted by frequency | none | + +## 4. Grading build tasks + +1. **Capture.** When a session ends, the runner writes `session.diff`: + `git add -A` and `git diff --cached HEAD`, leaving out + `.flatbread-proof/`, `.claude/`, `AGENTS.md`, `CLAUDE.md`, + `flatbread.config.js` and the runner's own files. It then deletes the + clone to save disk. +2. **Apply.** The diff is applied with `git apply --3way` to a pristine + worktree of the none variant at `p4/grade//`, which is reset and + cleaned before each session. So every arm is graded on byte-identical + code. If the diff does not apply, every check fails and `apply_failed` + is recorded. +3. **Restore.** Everything the bed tracks under `test/` (test files, + `helpers.ts`, fixtures) is restored from the pristine tree, with renames + seen as a deletion plus an addition. An agent that edited an existing + test is flagged `edited_existing_tests`, one that edited a helper or + fixture `edited_test_support`, and the originals still decide + regressions. +4. **Run.** Hidden tests are copied in only now, to `test/_hidden/`. Three + separate `node --experimental-strip-types --test` runs follow, each with + a 120 s timeout, with no network (`unshare -n`, where the machine allows + it) and no proxy or session variables: + - **Regressions:** the original suite. + - **Acceptance:** `acceptance.test.ts`, the behaviour asked for. + - **Constraints:** `constraints.test.ts` plus the global invariants. + `global.test.ts` runs on every task: a document saves, loads and saves + the same text, and plugin API v1 keeps its members and shortcut table + (both stated in the code). `global-rules.test.ts` runs on proof-only + and stale tasks only: every shipped migration step returns what it + always returned, and every palette command makes at most one history + entry per run. Those are R3 and R1, which only Proof states, so a + control or code-visible task is never graded on them. + The agent's own new tests run as well and are recorded + (`agent_tests_pass`) but not scored. `tsc --noEmit`, from the repo's + node_modules, is recorded as `typecheck` and is not scored. +5. **Report per task.** `acceptance`, `constraints`, `regressions` and + `all_green` (all three pass), plus the failing test names. + +**What each constraint test checks, by category.** Proof-only: the R rule +(for example, one `undo()` reverts every node in B-P1; a `JSON.stringify` +round-trip of the recorded command replays the same result in B-P2). +Code-visible: the V rule. Stale: the code's truth (a group's own opacity +changes and its children's opacity does not; rotation is in radians; the +frozen `DEFAULT_KEYMAP` snapshot is unchanged; `parseLength` behaves as +before on legacy strings). Control: global invariants only. + +**Offline gate** (`scripts/gate.mjs`, free, run before any paid session; +it must pass for every task): + +- The base tree fails acceptance and passes regressions and constraints. +- The reference patch passes all three. +- The naive patch passes acceptance and regressions, and fails only the + test file its category predicts. For stale tasks, the naive patch is the + "follow Proof" solution. B-S3 and B-S4 are the exception: their naive + patch fails acceptance, which is the predicted failure there. For + control tasks, the naive patch fails regressions only, and a second + "Proof-flavoured" reference that also keeps every R rule must pass + everything, which shows the constraint tests do not reward style. +- `leakWords` for each R rule appear nowhere in the none variant's code, + tests, docs or commit messages. The phrases for each V rule do appear. +- `featureWords` appear nowhere in Proof. +- The two variants' code trees are identical. + +## 5. Grading plan tasks + +- **Rubric.** Each plan task has a rubric in `p4/tasks//rubric.json`: + 4 to 6 must-include points and 2 to 4 must-avoid points, each a single + checkable claim (for example "edits to an instance override are + undoable in one step", or "must avoid: storing expand state on nodes in + the file"). Rubrics may name code modules, because plans will. +- **Blinding.** Before judging, `PLAN.md` is stripped of record ids (with + or without their hash suffix, and as qualified keys such as + `model.dec-group-nostyle`), `./flatbread proof` commands, `.flatbread-proof` + paths, the words "Proof" (not "future-proof" or "proof of") and "project + memory", phrases such as "the recorded decision" or "decision record", + and "Effort" and "compaction". Each is cut out together with the + parenthesis, bracket or lead-in ("per …", "see …", "… says:") that held + it, and nothing marks the cut, so a Proof-arm plan cannot be told from a + `none` plan by its markers and keeps the rest of each line. The judge is + told to score content only, whatever its source. +- **Judge.** A fixed judge, `claude-opus-5-5` at `--effort high` with no + tools, gets the stripped plan, the request and the rubric on stdin. It + returns JSON with met or violated for each point and a short quote. It + runs twice per plan. A point counts when both runs agree; when they + disagree, the point counts as half and is listed. +- **Score.** `(includes met + avoids not violated) / (|I| + |A|)`, from 0 + to 1. Must-avoid violations are also reported on their own as plan + constraint violations. +- **Check.** `scripts/judge.mjs --prepare-only` writes the judge inputs + with no paid call. The rubric gate (`judge.mjs --gate`) calls the judge, + so it runs in the pilot, before the full run. Each plan task has three + hand-written plans: a good one, which must score at least 0.8; a trap one + that walks into the stale or rule trap, which must score at most 0.5; + and a plausible one, a realistic plan written from the code alone as a + `none` session would, whose score is reported. On a proof-only task a + plausible plan that scores as high as the good one means the rubric does + not reward what only Proof says, and the rubric is revised before the + full run (one of the pilot's allowed changes). Control rubrics reward + only what the request and the code ask for, never an R rule. + +## 6. Arms + +The code and the request are identical in every arm. The whole task text +goes to `claude -p` on stdin, never as a file to read, because a pasted +brief or dump is larger than Linux's 128 KB limit on one argument and +larger than what `Read` returns. Every session is a fresh `claude -p` with +these settings: + +- `--model` and `--effort` recorded from the init event. +- `--strict-mcp-config` with no MCP servers. +- `--setting-sources project`, so a skill in the clone loads. +- Tools Bash, Read, Write, Edit, Glob, Grep and Skill in every arm. +- No session persistence and a 1,500 s timeout. + +Each session gets its own `git clone --no-hardlinks` with the remote +renamed and reflogs removed, as `prepareClone` does. The runner forks +`proof-recall/scripts/run.mjs` and keeps `--prepare-only`. +`FLATBREAD_PROOF_SERVER` is never set. Reads run locally against the +clone's own graph. + +| Arm | Clone | Added to the task text | +| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `none` | none variant; `AGENTS.md` (tests, style) | nothing | +| `skill` | proof variant; `.claude/skills/proof/` (a copy of `.agents/skills/proof/`); `./flatbread` wrapper; `AGENTS.md` plus one line: "Project memory (decisions, constraints, findings) lives in Proof; read it with `./flatbread proof …` (see the proof skill)"; `CLAUDE.md` = `@AGENTS.md` | nothing | +| `brief` | as `skill` | the output of `proof brief ` for each of the 7 Efforts, under "Project memory briefs" | +| `find` | as `skill` | "Before planning, run `./flatbread proof find --all`, then `proof get` any hit that bears on the request." | +| `findcode` | as `skill` | the `find` text plus: "Before editing a file, run `./flatbread proof find --path `; before relying on a Decision, run `proof find --refs all`." | +| `dump` | as `skill` | every live record body (all kinds, compactions included), grouped by Effort | +| `dump10x` | as `skill`, on the 10x graph | as `dump`, 1,074 live records of 1,547 (the burden ceiling; an extra arm, not a primary comparison) | + +`none` has no `CLAUDE.md` line about Proof, no skill, no wrapper, no +config and no node_modules link. The proof arms link node_modules (for the +CLI) through `.git/info/exclude`. Every arm's task text ends with the same +closing line: for build tasks "Implement it, add tests, and run the test +suite", and for plan tasks "Write PLAN.md in the repository root; do not +change code." + +## 7. Metrics and analysis + +- **Per session.** `all_green`, `acceptance`, `constraints` and + `regressions`; the number of failing constraint tests; plan score and + must-avoid hits. A constraint pass is counted only in a session that + also passed acceptance (`constraints`); the plain count, which an empty + diff passes, is kept as `constraints_raw` for reference. Cost comes from + `proof-recall/scripts/collect.mjs`: first-turn tokens (system prompt + plus task text, so the pasted brief or dump shows here), context tokens + (every fresh token, first turn included), read tokens (context tokens + without the first turn), processed tokens (context plus cache reads), + tool calls and turns. Wall time comes from the result event. Also + recorded: Proof reads (count of `./flatbread proof` calls), Proof writes + (any mutation), direct reads of `.flatbread-proof/` files, searches whose + results showed Proof files (`proof_search_exposure`: a repository-wide + Grep reaches records in every Proof arm, `skill` included), whether the + session opened a cited path, and `apply_failed`, `edited_existing_tests`, + `edited_test_support`, `timed_out` and `api_error`. +- **Runs.** Build tasks run 3 times per arm, plan tasks twice; `dump10x` + runs once. +- **Primary endpoints.** All-green on the proof-only build tasks (H1, + 6 tasks × 3 runs = 18 pairs per arm) and on the stale build tasks (H3, + 4 × 3 = 12 pairs). Each of the five arms (`skill`, `brief`, `find`, + `findcode`, `dump`) is compared with `none` in each of the two + categories by a two-sided exact sign test over the discordant + (task, run) pairs; the ten p values are Holm-corrected together at + α = 0.05 (`holm_primary` in `compare.mjs`). The stricter task-level sign + test, which counts each task once as in `proof-recall/scripts/compare.mjs`, + is reported beside each. `dump10x` is compared with `dump`, without + correction. +- **Secondary.** All-green pooled over all 18 build tasks, Holm-corrected + across the five arms (`holm_all_green`); it mixes categories whose + predicted effects run in opposite directions, so it is read only with + the category results beside it. Code-visible and control all-green + (H2, H4). Plan score, compared with a paired Wilcoxon signed-rank test + per arm against `none`. Context tokens, processed tokens and wall time + per arm, by median and paired ratio (H4). Every result is also reported + per category. +- **Leakage controls.** A session is flagged and excluded, together with + its pair across arms for that task and run, when any of these holds: + `left_folder` (a file tool, Glob, Grep or shell path outside the clone; + heredoc bodies and inline scripts count only for absolute paths under a + protected folder, so code written through the shell is not a leak); + any path under `p4/` or the benchmark source appears in a tool input + (`hidden_read` for the task, hidden and grading folders and the gate + log); or a `none` session found Proof by another route, which should be + impossible. The report gives flag counts by arm. Sessions that time out + stay in and count as failures. A session that ends in an API error is + not a task failure: transient errors are retried (up to 3 attempts), + and a (task, run) with any other API error is dropped from every arm and + counted per arm (`compare.mjs --errors fail` keeps them as failures, to + read a context overflow in `dump10x` as burden). + +## 8. Cost + +Rates are first-party API rates for `claude-haiku-5-5`: $0.10 per million +input tokens and $0.50 per million output tokens up to a 100K-token +prompt, and $0.50 and $2.50 above it. Cache reads cost 0.1× input and cache +writes 1.25× input. For `claude-sonnet-5-5`: $2 and $10, with cache reads +at $0.20. The judge is `claude-opus-5-5` at $4 and $20. + +The session sizes below are assumptions until the pilot measures them. +Past `proof-recall` runs read 0.3 to 0.6 M processed tokens per question, +and editing work costs more. The pasted text measured in prepare-only +runs is about 0.8 to 1.3 K characters for `none`, `skill`, `find` and +`findcode`, 24 K for `brief`, 75 K for `dump` and 481 K (about 120 K +tokens) for `dump10x`. + +| Session shape | Turns | Processed tokens | Cache writes | Output | Haiku cost | +| --------------------------------------- | ----- | ---------------- | ------------ | ------ | ---------- | +| build, `none` / `skill` / `find` | ~45 | ~2.0 M | ~120 K | ~20 K | ~$0.05 | +| build, `brief` (+~25 K fixed) | ~45 | ~3.1 M | ~150 K | ~20 K | ~$0.06 | +| build, `dump` (+~50 K fixed) | ~45 | ~4.3 M | ~180 K | ~20 K | ~$0.08 | +| build, `dump10x` (+~450 K, above 100 K) | ~45 | ~22 M | ~0.6 M | ~20 K | ~$1.50 | +| plan, any arm except `dump10x` | ~25 | ~1.0 M | ~80 K | ~8 K | ~$0.03 | + +- **Pilot** (one build task per category: B-P1, B-S3, B-V4, B-K1; the six + primary arms; 1 run; Haiku). 24 sessions, about $1.40 (range $1 to $4). + The rubric gate (24 hand-written plans, judged twice by Opus, 48 calls) + costs about $3. Two optional add-ons: `dump10x` on B-K1 (1 session, + about $1.50, to see that a 120 K-token prompt runs at all) and L-P1 in + the six arms (6 sessions and 12 judge calls, about $1, the only way to + exercise PLAN.md capture and the judge on real sessions before the full + run). The pilot must clear these gates before the full run: + + - every session finishes, with no API error left after retries; + - the init event lists the `proof` skill in the proof arms; + - `find` and `findcode` call `find` in at least 3 of 4 sessions; + - no leakage flag shows a harness bug; + - grading reproduces the gate's verdicts on the reference patches, + re-applied as session diffs; + - the rubric gate passes (good ≥ 0.8, trap ≤ 0.5); on a proof-only + task, a plausible plan that scores as high as the good one marks a + rubric to fix. + + The pilot may change prompts' wording, the timeout, the stripping rules + and, where the rubric gate shows it, a rubric's points, and nothing + else. Changes are logged below. + +- **Full run on Haiku** (18 build tasks × 6 arms × 3 runs = 324 sessions + and 8 plan tasks × 6 arms × 2 runs = 96, plus `dump10x` once per task, + 26). The 420 primary sessions cost about $20 to $28 and the 26 `dump10x` + sessions about $30 to $40. Judging covers 104 plans, twice each, for + about $14. The total is about $65 to $85. At 8 parallel jobs and about + 8 minutes per session, that is about 7.5 hours of wall time. +- **Sonnet subset** (the 8 build tasks P1, P3, P5, P7, V1, S1, S3 and K1; + the arms `none`, `skill`, `find` and `brief`; 1 run). 32 sessions at + about $0.90 each, for about $30. It is read only as a check on the + direction of the Haiku result. +- **Disk.** Clones are about 5 MB each and are deleted after capture. What + is kept for each session is `session.diff`, `PLAN.md`, the gzipped + transcript (about 0.3 MB) and the grade JSON, roughly 160 MB for the + full run. + +## Layout + +`benchmarks/proof-build/` holds: + +- `spec/history.mjs` and `spec/proof.mjs`: the bed's commits and records. +- `spec/filler.mjs`: the 10x filler. +- `tasks//`: each task's `task.json`, and for plan tasks `rubric.json` + and `plans/`. +- `scripts/` (`gen-bed.mjs`, `check-bed.mjs`, `gate.mjs`, `run.mjs`, + `grade.mjs`, `judge.mjs`, `compare.mjs`, with `arms.mjs`, `inject.mjs` and + `common.mjs`), reusing `proof-recall`'s `collect.mjs` and its sign test. + +`p4/` (outside the repo) holds: + +- `bed/`, `bed-none/`, `bed-x10/` and their key maps +- `hidden//` (hidden tests and patches) and `hidden/_global/` +- `tasks//` (assembled by `gate.mjs`) +- `grade/` +- `runs/` + +## Change log + +- 2026-10-20: pre-registered. +- 2026-10-20, task authoring, before any paid session: + - **Bed tests relaxed.** `test/plugin.test.ts` now pins the first three + plugin events instead of the whole list, since Proof allows new v1 + events and the old pin made every B-P5 solution fail regressions. + `test/persist.test.ts` checks the restored zoom and collapsed sections + field by field instead of comparing the whole view state, which made + every B-P6 solution that adds to the view state fail. The beds were + rebuilt (tips: bed `7a02228`, bed-none `ad36fdf`, bed-x10 `beda051`) + and `check-bed.mjs` passes. + - **Gists changed where the bed made the planned one useless.** B-P1 is + now Rename layers (numbered names, one undo), because field commits + already share a gesture id, so a corner-radius field could not break + R1. B-S1 is now number keys that set opacity, because the Opacity field + at the tip already handles groups, so the base tree passed the planned + task. B-K3 is now Distribute with a gap, because a layers filter needs a + facade method the prompt would have to name. B-P2 drops "continues the + same offset" (Mod+D on the copies moves them another 10 px). B-P3's + field is Layout grid (Off or a size). + - **Naive patches.** For B-S2 the Proof-following patch (snap and store + degrees) fails acceptance as well as constraints, because the Rotation + field shows the stored number read as radians; B-S4's fails acceptance + only, as planned. Control naive patches are plausible solutions that + break an existing test, so they fail regressions. No second + "Proof-flavoured" control reference was written. + - **Constraint tests** check their rule only where the build has the + feature, so the base tree passes them. + - **Global invariants** (`_global/global.test.ts`): every shipped migration + step returns what it returned at the tip on fixed inputs (including a + version 1 file with unit strings); a document saves, loads and saves the + same text (instead of the version 5 fixture round-trip, since a correct + schema change bumps the version); every palette command makes at most one + history entry per run, with and without input, over four selections; and + plugin API v1 keeps its members and its shortcut table. + - **Layout.** `task.json`, `rubric.json` and the good and trap plans are in + `benchmarks/proof-build/tasks/`; hidden tests and patches are in + `p4/hidden//`; `scripts/gate.mjs` (the `check.mjs` of §4) assembles + `p4/tasks/` and runs the offline gate. +- 2026-10-21, after the design and harness reviews, before any paid + session: + - **H1 is scored on all-green.** Constraint tests skip when the feature + is missing, so counting constraint passes alone let a session that + built nothing win. A constraint pass now counts only with acceptance, + and H1 reads all-green. + - **Primary endpoints per category.** The pooled all-green mixes + proof-only tasks (predicted help) with stale ones (predicted harm), so + proof-only and stale all-green are now co-primary, five arms each, + Holm-corrected together over the ten tests; the pooled result is + secondary. Build tasks run three times instead of twice. + - **Proof-only tasks that did not need Proof.** B-P2 (Duplicate) and B-P4 + (hold an arrow key) are now code-visible, as B-V4 and B-V5: the `Json` + payload type, `commands.replay` and `groupNodes` show R2, and + `KeyEvent.repeat`, the unused `release()` hook, `ctx.newGesture()` and + the test "commands without a gesture id stay apart however close in + time" show R4. B-V4's naive patch is now a realistic mistake (selecting + the copies after the batch). B-V5's prompt no longer echoes a record + ("separate presses stay separate"). Two proof-only tasks replace them: + B-P7 (Copy style and Paste style, R1) and B-P8 (old files that name + colours, R3). + - **B-P3 could be passed without Proof.** A load-time default with no new + step passed every test, and the accepted record + `persist.dec-defaults-on-load` said "Migrations do not have to add + every field", against `persist.con-migrations-frozen`. The record now + says load defaults cover files from before versions existed and are + not how the format gains a field; a new constraint test requires a + newer saved version reached by a step from version 5; the naive patch + is now the load-time default alone. The L-P1 and L-P3 rubrics, which + ask for a new step, now agree with Proof. + - **B-P6 gave its answer away.** The prompt no longer mentions other view + settings, its acceptance test for them is gone (the bed's own tests + cover them), and `persist.dec-viewstate-sidecar` no longer lists + "which groups the layers panel shows expanded". + - **B-P5: Proof and code disagreed.** `plugin.dec-ids-only-events` now + says events carry ids and plain copied values, never nodes, instead of + "a plugin that needs more asks for a capability". The plugin event + payloads are now frozen where they are emitted (commit c33, "Remove + getNode from the plugin API"), as `plugin.con-frozen-snapshots` says; + the freeze is shallow, so deep freezing and copies stay Proof-only. The + naive patch is now plain copies in a top-level frozen payload, which is + what copying the existing emitters gives. + - **Prompts that steered.** B-S1 no longer says "exactly what typing that + percentage into the Opacity field does, … groups included"; B-S3 no + longer points at Mod+Shift+L, which sits in both the canvas bindings + and `DEFAULT_KEYMAP`. + - **Rules graded where they cannot be seen.** The global invariants are + split: R1 and R3 (`global-rules.test.ts`) run only with proof-only and + stale tasks; stable saving and plugin API v1 (`global.test.ts`) run + with every task. + - **Control plan rubrics rewarded Proof rules.** L-K1 and L-K2 no longer + have points about the document file, history or the command bus; their + trap plans match the new points. + - **Rubric gate.** Each plan task gains `plans/plausible.md`, a realistic + plan written from the code alone. `judge.mjs --gate` judges good, trap + and plausible plans; it runs in the pilot, since it calls the judge. + - **Bed.** `test/persist.test.ts`'s fixture round-trip now checks that + every field of the version 5 fixture survives a save, rather than the + whole file, so a correct schema change does not fail regressions and + grading can restore fixtures. The filler topic "colour names" is now + "colour pickers", and the new tasks' feature words are banned in + Proof. Rebuilt tips: bed `598221b`, bed-none `bdb60f4`, bed-x10 + `06b0e6d`; `check-bed.mjs` passes. + - **Harness.** The leak scan no longer flags `..`, `/` or `//` inside + heredoc bodies and inline scripts, and watches the whole `p4/` folder + (hidden tests, gate log) in Bash too. Cost reports first-turn and + context tokens, so the pasted brief and dump count toward H4. Plan + blinding cuts Proof wording with its clause instead of leaving `[ref]` + markers that only Proof-arm plans would carry. API errors are retried + or reported apart. Renamed test files and edited helpers or fixtures + no longer slip past grading. Grading runs without network where + `unshare -n` works. `run.mjs` defaults match the `p4/` layout, and its + printed grade command passes `--bed`. +- 2026-10-21, offline gate before the pilot, before any paid session: + - **Beds rebuilt from scratch** match the previous builds object for + object (commit and tree hashes, key maps, working trees), so the build + is deterministic. `check-bed.mjs` passes 33 checks and the suite 132 + of 132. + - **Blinding touched plans it should not have.** `stripPlan` collapsed + indentation, dropped `()` after function names and removed trailing + spaces in every plan, so 9 of the 24 hand-written plans changed with + no Proof wording in them. It now edits only lines that held a cut, so a + plan with no Proof wording comes out byte for byte the same. It also + cut `an ` out of words such as "ran" ("I ran `./flatbread proof find …`" became "I r and read …"), left a dangling "as." where "as the + project memory records" was cut, and left empty backticks round a cut + id. A sentence that only reports running a Proof command now goes + whole, a lead-in left pointing at nothing goes, a code span holding + only cut ids goes, and a line whose opening words were cut starts + upper case. + - **Pilot tasks.** The pilot now takes one build task per category + (B-P1, B-S3, B-V4, B-K1) in the six primary arms, 24 sessions; L-P1 + and `dump10x` are optional add-ons. The `dump10x` arm text gives its + real size: 1,074 live records of the 1,547 record files. diff --git a/benchmarks/proof-build/README.md b/benchmarks/proof-build/README.md new file mode 100644 index 00000000..ed2f1b2a --- /dev/null +++ b/benchmarks/proof-build/README.md @@ -0,0 +1,248 @@ +# Proof build benchmark + +When an agent builds a feature from a product-level request ("the +properties panel should also…"), does having Proof records in its context +make the work better, worse, or just more expensive? The pre-registered +design, hypotheses and change log are in [DESIGN.md](DESIGN.md). + +## What it measures + +The test bed is Tessera core, a small zero-dependency TypeScript design +editor with 84 commits of history. In the proof variant, 7 Efforts of +records written the way real agents write them (Decisions with rejected +alternatives, Constraints, Findings, Risks, code Citations) sit in the +graph. The none variant has the same code and no Proof at all. The 10x +variant adds 9 filler Efforts, about ten times the records. + +There are 26 tasks: 18 build tasks graded by hidden tests and 8 plan tasks +scored by a blind rubric. Each task belongs to one category: + +| Category | What makes a solution right | Build | Plan | +| ------------ | ----------------------------------------------------- | ----- | ---- | +| proof-only | a rule that only Proof states | 6 | 3 | +| code-visible | a rule Proof states and the code also shows | 5 | 1 | +| stale | the code, where Proof is out of date | 4 | 2 | +| control | nothing about Proof; measures the cost of carrying it | 3 | 2 | + +Every session gets the same code and the same request. The arms differ in +how Proof reaches the agent: + +| Arm | Proof in the session | +| ---------- | -------------------------------------------------------------- | +| `none` | none: no records, no trailers, no CLI, no skill | +| `skill` | available through the proof skill and `./flatbread proof` | +| `find` | as `skill`, plus "run `proof find` on the request first" | +| `findcode` | as `find`, plus `find --path ` before edits and `--refs` | +| `brief` | as `skill`, plus `proof brief` of every Effort pasted in | +| `dump` | as `skill`, plus every live record pasted in | +| `dump10x` | `dump` on the 10x graph (the burden ceiling, one run) | + +A build session is all-green when its acceptance, constraint and +regression tests all pass. The primary endpoints are all-green on the +proof-only tasks (does Proof help?) and on the stale tasks (does stale +Proof hurt?), each arm against `none`, with Holm correction. Cost (first +turn, context and processed tokens, turns, wall time) answers the burden +question. Leakage flags drop sessions that read outside their clone or +reach the hidden tests. + +## Build the beds + +Beds are built outside the repository, in `` (the key maps hold the +answers). Builds are deterministic: a rebuild gives the same hashes. The +hidden tests and the reference and naive patches are committed in `hidden/` +since the first full run; copy them to `/hidden` before running, so a +session never needs a path into this repository, and pass `--out` to +`gate.mjs` (it refuses to rebuild `tasks/` in place). `grade.mjs` flags any +session that reads `hidden/` or `tasks/` here. + +```sh +node scripts/gen-bed.mjs --out /bed +node scripts/gen-bed.mjs --variant none --out /bed-none +node scripts/gen-bed.mjs --scale 10 --out /bed-x10 +node scripts/check-bed.mjs --proof /bed --none /bed-none --x10 /bed-x10 +``` + +`--check-each` runs the bed's suite at every commit. + +## Offline gate (free) + +```sh +node scripts/gate.mjs --hidden /hidden --none /bed-none \ + --bed /bed --x10 /bed-x10 +``` + +It assembles `/tasks/`, runs the static checks (no feature word in +Proof, no Proof-only rule anywhere in the none bed, prompts name no file or +symbol), and grades each build task's base tree, reference patch and naive +patch: base fails acceptance, reference passes everything, naive fails the +group it is meant to fail. It exits 1 on any failure. + +## Run + +Prepare first and read what a session would see (no model is called): + +```sh +node scripts/run.mjs /runs --name r1 --bed /bed --prepare-only --ids B-P1 +``` + +Then run for real (`--arms`, `--ids`, `--only `, +`--jobs`, `--model`): + +```sh +node scripts/run.mjs /runs --name r1 --bed /bed --model claude-haiku-5-5 --jobs 8 +``` + +Each session gets a fresh clone and the whole task text on stdin. A rerun +skips finished sessions and retries transient API errors. + +## Grade, judge, compare + +```sh +node scripts/grade.mjs /runs --name r1 --tasks /tasks \ + --none /bed-none --bed /bed [--bed10x /bed-x10] +node scripts/judge.mjs /runs --name r1 --tasks /tasks [--dry-run] +node scripts/compare.mjs /runs --runs r1,r2,r3 --out /runs/compare.json +``` + +`grade.mjs` applies each session's diff to the none bed, restores the +bed's own tests, adds the hidden tests and runs them without network. +`judge.mjs` strips Proof ids and wording from each PLAN.md so the judge +cannot tell the arm, then scores it against the rubric twice; +`--dry-run` prints the blinded prompt. `compare.mjs` pairs each arm with +`none` by task and run. + +Before the full run, the rubric gate checks that the rubrics separate the +hand-written good, trap and plausible plans (it calls the judge): + +```sh +node scripts/judge.mjs --gate --tasks /tasks --out /gate-judge.json +``` + +## Pilot and full run + +The pilot is one build task per category in the six primary arms, once, +on Haiku: 24 sessions, about $1.40 (range $1 to $4), plus about $3 for +the rubric gate. With `P4` set to the `` folder: + +```sh +node scripts/run.mjs $P4/runs --name pilot --bed $P4/bed \ + --ids B-P1,B-S3,B-V4,B-K1 --model claude-haiku-5-5 --jobs 8 +node scripts/grade.mjs $P4/runs --name pilot --tasks $P4/tasks \ + --none $P4/bed-none --bed $P4/bed +node scripts/compare.mjs $P4/runs --runs pilot --out $P4/runs/pilot.json +node scripts/judge.mjs --gate --tasks $P4/tasks --out $P4/gate-judge.json +``` + +The full run is 420 primary sessions (build tasks 3 runs, plan tasks 2) +and 26 `dump10x` sessions, about 6 hours of wall time at 8 jobs: + +```sh +for r in r1 r2; do + node scripts/run.mjs $P4/runs --name $r --bed $P4/bed --model claude-haiku-5-5 --jobs 8 +done +node scripts/run.mjs $P4/runs --name r3 --bed $P4/bed --only build --model claude-haiku-5-5 --jobs 8 +node scripts/run.mjs $P4/runs --name r1 --bed $P4/bed --arms dump10x --model claude-haiku-5-5 --jobs 4 +for r in r1 r2 r3; do + node scripts/grade.mjs $P4/runs --name $r --tasks $P4/tasks \ + --none $P4/bed-none --bed $P4/bed --bed10x $P4/bed-x10 + node scripts/judge.mjs $P4/runs --name $r --tasks $P4/tasks +done +node scripts/compare.mjs $P4/runs --runs r1,r2,r3 --out $P4/runs/compare.json +``` + +## Results: Haiku, full run (2026-10-21) + +18 build tasks × 3 runs and 8 plan tasks × 2 runs in each of the six primary +arms, plus `dump10x` once per task: 446 sessions on `claude-haiku-5-5`, after +a 30-session pilot. Per-session rows are in `results/haiku-sessions.json` +(no transcripts); the `compare.mjs` output is `results/haiku-compare.json`. +No session timed out or hit an API error. 23 sessions were flagged for leaving +their folder (scratch files, `cd /tmp`, reading the prettier binary), none for +reading hidden tests, and no `none` session found Proof. The tables keep the +flagged sessions; dropping their pairs gives the same picture. + +**Build tasks: all-green, of 54 per arm** (proof-only 18, code-visible 15, +stale 12, control 9; `dump10x` ran each task once: 6, 5, 4 and 3): + +| Arm | proof-only | code-visible | stale | control | all | context tokens | processed tokens | +| ---------- | ---------- | ------------ | ------ | ------- | ------------ | -------------- | ---------------- | +| `none` | 17 | 12 | **12** | 6 | **47** (87%) | 42k | 594k | +| `skill` | 18 | 9 | 8 | 6 | 41 (76%) | 51k | 781k | +| `find` | 18 | 7 | 9 | 6 | 40 (74%) | 46k | 668k | +| `findcode` | 18 | 8 | 9 | 6 | 41 (76%) | 53k | 798k | +| `brief` | 18 | 6 | 7 | 6 | 37 (69%) | 60k | 901k | +| `dump` | 18 | 6 | 6 | 6 | 36 (67%) | 78k | 1,197k | +| `dump10x` | 6 of 6 | 2 of 5 | 2 of 4 | 2 of 3 | 12 of 18 | 255k | 3,887k | + +- **One stale Decision did almost all the harm.** It says shortcuts are + registered in `keys/keymap.ts`. A later refactor moved registration to the + scope bindings and left that file as the frozen table plugins read, with no + Proof update. Every task that adds a shortcut met it: B-S3 (`none` 3 of 3, + every Proof arm 0 of 3), B-V4 (Proof arms 0 to 1 of 3) and B-V3 (0 to 2 of + 3). In B-S1 a second stale record, on group opacity, also caught `dump` (0 of + 3). The sessions edited the frozen table and broke plugin API v1. +- **A stale-anchor note did not stop it.** `findcode` sessions read + `#registerBinding renamed to legacyAction in bc1c5b5; 4 commits touched it since` and still followed the accepted Decision. +- **More forced Proof, more harm and more cost.** All-green falls from 74–76% + (`skill`, `find`, `findcode`) to 69% (`brief`) and 67% (`dump`), while + processed tokens rise to 1.1–1.3×, 1.5× and 2.0× those of `none`. +- **Proof-only tasks showed no gain.** `none` passed 17 of 18. Haiku inferred + one undo step per gesture, unedited shipped migrations and frozen plugin + payloads from the patterns already in the code, so these tasks cannot yet + show a benefit. +- **Two tasks fail in every arm.** B-K2 breaks a colour-parsing regression and + B-V2 fails exact round-trip scaling, so neither says anything about Proof. +- **Statistics.** Pooled all-green against `none`, pair level and + Holm-corrected: `dump` p = 0.059, `brief` 0.086, the others 0.21 to 0.29. + The pre-registered primary tests (proof-only and stale, Holm-corrected) are + not significant (best: stale `dump`, 0.31). Counting each task once gives + 1 task against 4 (p = 0.375): the harm sits in a few tasks that share one + stale record, so pair-level p-values overstate it. + +**Plan tasks: mean rubric score** (proof-only 6, code-visible 2, stale 4 and +control 4 plans per arm; `dump10x` half as many): + +| Arm | proof-only | code-visible | stale | control | all | +| ---------- | ---------- | ------------ | ----- | ------- | ---- | +| `none` | 0.90 | 0.57 | 1.00 | 1.00 | 0.91 | +| `skill` | 0.91 | 0.86 | 0.94 | 0.93 | 0.91 | +| `find` | 0.89 | 0.79 | 1.00 | 0.93 | 0.91 | +| `findcode` | 0.91 | 0.86 | 0.93 | 1.00 | 0.93 | +| `brief` | 0.98 | 0.79 | 0.91 | 0.96 | 0.93 | +| `dump` | 1.00 | 0.86 | 0.76 | 0.96 | 0.91 | +| `dump10x` | 0.96 | 0.71 | 0.52 | 0.93 | 0.81 | + +Plans show both edges. Pasted Proof lifts proof-only plans (`dump` 1.00 and +`brief` 0.98, against 0.90) and drags down stale ones (`dump` 0.76 and +`dump10x` 0.52, against 1.00), so overall they cancel. No plan difference is +significant (Wilcoxon p ≥ 0.30). + +So on this bed Proof in context never raised build quality, and an +out-of-date record lowered it wherever its area was touched, most when Proof +was pasted in. A plan gains from rules it is told, but the code already +teaches a builder most of them. The changes this points to are a read that +marks a Decision itself as suspect when the code it rests on has moved, and +proof-only tasks whose rules the code does not hint at. + +## Layout + +| Path | What it holds | +| ----------------------- | ----------------------------------------------------------------------------- | +| `spec/app/` | the bed's code at the tip: sources, tests, fixtures, docs | +| `spec/history.mjs` | the 84 commits and the change each one makes | +| `spec/versions/` | whole earlier files for the two changes too large for text pairs | +| `spec/proof.mjs` | the 7 Efforts and every Proof mutation, by commit | +| `spec/filler.mjs` | the 9 filler Efforts of the 10x graph | +| `spec/banned.mjs` | words no record or commit may hold, so Proof never names a task | +| `tasks//` | `task.json`; plan tasks also `rubric.json` and `plans/` | +| `scripts/gen-bed.mjs` | builds a bed | +| `scripts/check-bed.mjs` | checks built beds against each other and the design | +| `scripts/gate.mjs` | the offline gate | +| `scripts/run.mjs` | runs or prepares sessions (`arms.mjs` arm texts, `inject.mjs` brief and dump) | +| `scripts/grade.mjs` | grades sessions, or one patch | +| `scripts/judge.mjs` | blinds and judges plans; the rubric gate | +| `scripts/compare.mjs` | compares each arm with `none` | + +Hidden tests and the reference and naive patches are in +`/hidden//`, never in this repository. `FLATBREAD_PROOF_SERVER` +must stay unset; Proof reads run locally against each clone's own graph. diff --git a/benchmarks/proof-build/hidden/B-K1/acceptance.test.ts b/benchmarks/proof-build/hidden/B-K1/acceptance.test.ts new file mode 100644 index 00000000..25f7c094 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K1/acceptance.test.ts @@ -0,0 +1,74 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const field = (e: Editor, label: string) => + e.panel + .view() + .sections.flatMap((s) => s.fields) + .find((f) => f.label === label); + +function oneText(text: string) { + const e = createEditor(); + const t = e.create('text', { text, width: 400 }); + e.selection.set([t]); + return { e, t }; +} + +test('a text layer shows its character and word counts', () => { + const { e } = oneText('Hello brave new\nworld'); + assert.equal(e.panel.value('Characters'), '20'); + assert.equal(e.panel.value('Words'), '4'); +}); + +test('spaces count as characters; line breaks do not; runs of spaces make one gap', () => { + const { e } = oneText(' two words '); + assert.equal(e.panel.value('Characters'), '15'); + assert.equal(e.panel.value('Words'), '2'); + const empty = oneText(''); + assert.equal(empty.e.panel.value('Characters'), '0'); + assert.equal(empty.e.panel.value('Words'), '0'); + const lines = oneText('a\n\nb\n'); + assert.equal(lines.e.panel.value('Characters'), '2'); + assert.equal(lines.e.panel.value('Words'), '2'); +}); + +test('both fields are read-only', () => { + const { e } = oneText('Hi there'); + assert.equal(field(e, 'Characters')?.readonly, true); + assert.equal(field(e, 'Words')?.readonly, true); + assert.equal(e.panel.edit('Words', '10').ok, false); + assert.equal(e.panel.value('Words'), '2'); + assert.equal(e.history.size, 0); +}); + +test('the counts follow edits to the text', () => { + const { e } = oneText('one'); + e.panel.edit('Text', 'one two three'); + assert.equal(e.panel.value('Words'), '3'); + assert.equal(e.panel.value('Characters'), '13'); + e.history.undo(); + assert.equal(e.panel.value('Words'), '1'); +}); + +test('several text layers show Mixed when their counts differ', () => { + const e = createEditor(); + const a = e.create('text', { text: 'same size' }); + const b = e.create('text', { text: 'size same' }); + const c = e.create('text', { text: 'longer text here' }); + e.selection.set([a, b]); + assert.equal(e.panel.value('Characters'), '9'); + assert.equal(e.panel.value('Words'), '2'); + e.selection.set([a, c]); + assert.equal(e.panel.value('Characters'), 'Mixed'); + assert.equal(e.panel.value('Words'), 'Mixed'); +}); + +test('other layers do not show the counts', () => { + const e = createEditor(); + const r = e.create('rect'); + e.selection.set([r]); + assert.equal(e.panel.value('Characters'), undefined); + assert.equal(e.panel.value('Words'), undefined); +}); diff --git a/benchmarks/proof-build/hidden/B-K1/naive.patch b/benchmarks/proof-build/hidden/B-K1/naive.patch new file mode 100644 index 00000000..85e5a0fb --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K1/naive.patch @@ -0,0 +1,55 @@ +diff --git a/src/panel/properties.ts b/src/panel/properties.ts +index 4623d29..b4ee8b8 100644 +--- a/src/panel/properties.ts ++++ b/src/panel/properties.ts +@@ -11,6 +11,7 @@ import { fillSection } from './sections/fill.ts'; + import { strokeSection } from './sections/stroke.ts'; + import { textSection } from './sections/text.ts'; + import { appearanceSection } from './sections/appearance.ts'; ++import { textInfoSection } from './sections/text-info.ts'; + + export interface FieldView { + label: string; +@@ -38,6 +39,7 @@ export const DEFAULT_SECTIONS: readonly SectionDescriptor[] = [ + fillSection, + strokeSection, + textSection, ++ textInfoSection, + ]; + + /** +diff --git a/src/panel/sections/text-info.ts b/src/panel/sections/text-info.ts +new file mode 100644 +index 0000000..d832b34 +--- /dev/null ++++ b/src/panel/sections/text-info.ts +@@ -0,0 +1,29 @@ ++import type { SectionDescriptor } from '../schema.ts'; ++import { field } from '../schema.ts'; ++import type { TextNode } from '../../model/types.ts'; ++import { isText } from '../../model/types.ts'; ++ ++/** Read-only counts for text layers. */ ++export const textInfoSection: SectionDescriptor = { ++ name: 'text-info', ++ title: 'Text info', ++ fields: [ ++ field({ ++ label: 'Characters', ++ kind: 'number', ++ readonly: true, ++ appliesTo: isText, ++ read: (node) => (node as TextNode).text.replace(/\n/g, '').length, ++ write: () => [], ++ }), ++ field({ ++ label: 'Words', ++ kind: 'number', ++ readonly: true, ++ appliesTo: isText, ++ read: (node) => ++ (node as TextNode).text.split(/\s+/).filter(Boolean).length, ++ write: () => [], ++ }), ++ ], ++}; diff --git a/benchmarks/proof-build/hidden/B-K1/reference.patch b/benchmarks/proof-build/hidden/B-K1/reference.patch new file mode 100644 index 00000000..686874e6 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K1/reference.patch @@ -0,0 +1,56 @@ +diff --git a/src/model/text.ts b/src/model/text.ts +index 38b5335..89abef2 100644 +--- a/src/model/text.ts ++++ b/src/model/text.ts +@@ -54,6 +54,16 @@ export function autoHeight( + return lineCount(node) * node.fontSize * node.lineHeight; + } + ++/** Characters in `text`, spaces included and line breaks left out. */ ++export function characterCount(text: string): number { ++ return [...text.replace(/\r?\n/g, '')].length; ++} ++ ++/** Runs of non-space characters in `text`. */ ++export function wordCount(text: string): number { ++ return text.split(/\s+/).filter(Boolean).length; ++} ++ + export function isBlank(text: string): boolean { + return text.trim().length === 0; + } +diff --git a/src/panel/sections/text.ts b/src/panel/sections/text.ts +index 65960b1..8681237 100644 +--- a/src/panel/sections/text.ts ++++ b/src/panel/sections/text.ts +@@ -3,7 +3,7 @@ import { field } from '../schema.ts'; + import { command } from '../../commands/types.ts'; + import type { TextNode } from '../../model/types.ts'; + import { isText } from '../../model/types.ts'; +-import { autoHeight } from '../../model/text.ts'; ++import { autoHeight, characterCount, wordCount } from '../../model/text.ts'; + + function setText( + node: TextNode, +@@ -52,5 +52,21 @@ export const textSection: SectionDescriptor = { + write: (node, value) => + setText(node as TextNode, 'lineHeight', value as number), + }), ++ field({ ++ label: 'Characters', ++ kind: 'number', ++ readonly: true, ++ appliesTo: isText, ++ read: (node) => characterCount((node as TextNode).text), ++ write: () => [], ++ }), ++ field({ ++ label: 'Words', ++ kind: 'number', ++ readonly: true, ++ appliesTo: isText, ++ read: (node) => wordCount((node as TextNode).text), ++ write: () => [], ++ }), + ], + }; diff --git a/benchmarks/proof-build/hidden/B-K2/acceptance.test.ts b/benchmarks/proof-build/hidden/B-K2/acceptance.test.ts new file mode 100644 index 00000000..f869c319 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K2/acceptance.test.ts @@ -0,0 +1,62 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const r = e.create('rect'); + e.selection.set([r]); + return { e, r }; +} + +test('Fill takes three-digit shorthand, with or without #', () => { + const { e } = setup(); + assert.ok(e.panel.edit('Fill', '#f80').ok); + assert.equal(e.panel.value('Fill'), '#FF8800'); + e.panel.edit('Fill', '0aF'); + assert.equal(e.panel.value('Fill'), '#00AAFF'); +}); + +test('Fill takes six digits with or without #', () => { + const { e } = setup(); + e.panel.edit('Fill', '12ab9c'); + assert.equal(e.panel.value('Fill'), '#12AB9C'); + e.panel.edit('Fill', '#C0FFEE'); + assert.equal(e.panel.value('Fill'), '#C0FFEE'); +}); + +test('Fill takes rgb() with values from 0 to 255', () => { + const { e } = setup(); + e.panel.edit('Fill', 'rgb(255, 136, 0)'); + assert.equal(e.panel.value('Fill'), '#FF8800'); + e.panel.edit('Fill', ' RGB( 0,0,255 ) '); + assert.equal(e.panel.value('Fill'), '#0000FF'); + e.panel.edit('Fill', 'rgb(1,2,3)'); + assert.equal(e.panel.value('Fill'), '#010203'); +}); + +test('Stroke takes the same forms', () => { + const { e } = setup(); + e.panel.edit('Stroke', '#abc'); + assert.equal(e.panel.value('Stroke'), '#AABBCC'); + e.panel.edit('Stroke', 'rgb(16, 32, 48)'); + assert.equal(e.panel.value('Stroke'), '#102030'); +}); + +test('anything else is refused and changes nothing', () => { + const { e } = setup(); + e.panel.edit('Fill', '#123456'); + const size = e.history.size; + for (const bad of [ + 'rgb(300, 0, 0)', + 'rgb(1, 2)', + 'rgb(-1, 0, 0)', + 'orange', + '#ff88', + '#ggg', + 'rgb(1.5, 2, 3)', + ]) + assert.equal(e.panel.edit('Fill', bad).ok, false, bad); + assert.equal(e.panel.value('Fill'), '#123456'); + assert.equal(e.history.size, size); +}); diff --git a/benchmarks/proof-build/hidden/B-K2/naive.patch b/benchmarks/proof-build/hidden/B-K2/naive.patch new file mode 100644 index 00000000..19a046fe --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K2/naive.patch @@ -0,0 +1,40 @@ +diff --git a/src/model/style.ts b/src/model/style.ts +index 3ed3bac..79cb00a 100644 +--- a/src/model/style.ts ++++ b/src/model/style.ts +@@ -6,11 +6,32 @@ import { clamp } from '../util/math.ts'; + export const DEFAULT_FILL = '#D9D9D9'; + + const HEX6 = /^#?([0-9a-f]{6})$/i; ++const HEX3 = /^#?([0-9a-f])([0-9a-f])([0-9a-f])$/i; ++const RGB = /^rgb\(\s*(\d{1,3})\s*,\s*(\d{1,3})\s*,\s*(\d{1,3})\s*\)$/i; + +-/** Parses `#rrggbb` in any case. Returns `#RRGGBB`, or null. */ ++/** ++ * Parses `#rrggbb`, `#rgb` or `rgb(r, g, b)` in any case. Returns ++ * `#RRGGBB`, or null. ++ */ + export function parseColor(text: string): string | null { +- const match = HEX6.exec(text.trim()); +- return match ? `#${match[1].toUpperCase()}` : null; ++ const clean = text.trim(); ++ const match = HEX6.exec(clean); ++ if (match) return `#${match[1].toUpperCase()}`; ++ const short = HEX3.exec(clean); ++ if (short) ++ return `#${short ++ .slice(1) ++ .map((d) => d + d) ++ .join('') ++ .toUpperCase()}`; ++ const rgb = RGB.exec(clean); ++ if (!rgb) return null; ++ const channels = rgb.slice(1).map(Number); ++ if (channels.some((c) => c > 255)) return null; ++ return `#${channels ++ .map((c) => c.toString(16).padStart(2, '0')) ++ .join('') ++ .toUpperCase()}`; + } + + export function formatColor(color: string): string { diff --git a/benchmarks/proof-build/hidden/B-K2/reference.patch b/benchmarks/proof-build/hidden/B-K2/reference.patch new file mode 100644 index 00000000..bb4895ff --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K2/reference.patch @@ -0,0 +1,62 @@ +diff --git a/src/model/style.ts b/src/model/style.ts +index 3ed3bac..fef0f9d 100644 +--- a/src/model/style.ts ++++ b/src/model/style.ts +@@ -13,6 +13,35 @@ export function parseColor(text: string): string | null { + return match ? `#${match[1].toUpperCase()}` : null; + } + ++const HEX3 = /^#?([0-9a-f])([0-9a-f])([0-9a-f])$/i; ++const RGB = /^rgb\(\s*(\d{1,3})\s*,\s*(\d{1,3})\s*,\s*(\d{1,3})\s*\)$/i; ++ ++/** ++ * Reads a colour as people type it into a field: `#rrggbb` or `rrggbb`, ++ * the shorthand `#rgb` or `rgb`, or `rgb(r, g, b)` with 0–255 channels. ++ * Returns `#RRGGBB`, or null. ++ */ ++export function parseColorInput(text: string): string | null { ++ const clean = text.trim(); ++ const full = parseColor(clean); ++ if (full) return full; ++ const short = HEX3.exec(clean); ++ if (short) ++ return `#${short ++ .slice(1) ++ .map((d) => d + d) ++ .join('') ++ .toUpperCase()}`; ++ const rgb = RGB.exec(clean); ++ if (!rgb) return null; ++ const channels = rgb.slice(1).map(Number); ++ if (channels.some((c) => c > 255)) return null; ++ return `#${channels ++ .map((c) => c.toString(16).padStart(2, '0')) ++ .join('') ++ .toUpperCase()}`; ++} ++ + export function formatColor(color: string): string { + return color.toUpperCase(); + } +diff --git a/src/panel/fields.ts b/src/panel/fields.ts +index 2643eef..16b8f08 100644 +--- a/src/panel/fields.ts ++++ b/src/panel/fields.ts +@@ -2,7 +2,7 @@ import type { FieldDescriptor, FieldKind, FieldValue } from './schema.ts'; + import { parseExpression } from '../units/expr.ts'; + import { displayNumber } from '../units/precision.ts'; + import { formatAngle, formatLength, formatPercent } from '../units/format.ts'; +-import { formatColor, parseColor } from '../model/style.ts'; ++import { formatColor, parseColorInput } from '../model/style.ts'; + import { clamp, degToRad } from '../util/math.ts'; + + /** Reads what the user typed into a field. Returns null when it cannot. */ +@@ -23,7 +23,7 @@ export function parseFieldInput( + return percent === null ? null : percent / 100; + } + case 'color': +- return parseColor(text); ++ return parseColorInput(text); + case 'text': + return text; + } diff --git a/benchmarks/proof-build/hidden/B-K3/acceptance.test.ts b/benchmarks/proof-build/hidden/B-K3/acceptance.test.ts new file mode 100644 index 00000000..a24a97a7 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K3/acceptance.test.ts @@ -0,0 +1,80 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const xs = (e: Editor, ids: string[]) => ids.map((id) => e.doc.require(id).x); +const ys = (e: Editor, ids: string[]) => ids.map((id) => e.doc.require(id).y); + +function row() { + const e = createEditor(); + const a = e.create('rect', { x: 0, y: 0, width: 50, height: 50 }); + const b = e.create('rect', { x: 100, y: 0, width: 30, height: 50 }); + const c = e.create('rect', { x: 300, y: 0, width: 50, height: 50 }); + return { e, a, b, c }; +} + +test('Distribute horizontally with a gap lays the layers out from the leftmost', () => { + const { e, a, b, c } = row(); + e.selection.set([c, a, b]); + assert.ok(e.palette.run('Distribute horizontally', '20')); + assert.deepEqual(xs(e, [a, b, c]), [0, 70, 120]); +}); + +test('the order comes from the layers’ positions, not the selection', () => { + const { e, a, b, c } = row(); + e.doc.update(b, { x: 400 }); + e.selection.set([b, a, c]); + e.palette.run('Distribute horizontally', '10'); + assert.deepEqual(xs(e, [a, c, b]), [0, 60, 120]); +}); + +test('Distribute vertically with a gap lays them out from the topmost', () => { + const e = createEditor(); + const p = e.create('rect', { x: 0, y: 10, width: 50, height: 40 }); + const q = e.create('rect', { x: 0, y: 500, width: 50, height: 10 }); + const r = e.create('rect', { x: 0, y: 90, width: 50, height: 20 }); + e.selection.set([p, q, r]); + e.palette.run('Distribute vertically', '5'); + assert.deepEqual(ys(e, [p, r, q]), [10, 55, 80]); +}); + +test('a gap of 0 puts them edge to edge', () => { + const { e, a, b, c } = row(); + e.selection.set([a, b, c]); + e.palette.run('Distribute horizontally', '0'); + assert.deepEqual(xs(e, [a, b, c]), [0, 50, 80]); + e.palette.run('Distribute horizontally', '16'); + assert.deepEqual(xs(e, [a, b, c]), [0, 66, 112]); +}); + +test('layers inside a group move with it', () => { + const e = createEditor(); + const a = e.create('rect', { x: 0, y: 0, width: 50, height: 50 }); + const g = e.create('group'); + const inner = e.create( + 'rect', + { x: 200, y: 0, width: 40, height: 50 }, + { parent: g } + ); + const c = e.create('rect', { x: 500, y: 0, width: 50, height: 50 }); + e.selection.set([a, g, c]); + e.palette.run('Distribute horizontally', '10'); + assert.deepEqual(xs(e, [a, inner, c]), [0, 60, 110]); +}); + +test('a gap is one step to undo', () => { + const { e, a, b, c } = row(); + e.selection.set([a, b, c]); + e.palette.run('Distribute horizontally', '20'); + e.history.undo(); + assert.deepEqual(xs(e, [a, b, c]), [0, 100, 300]); +}); + +test('text that is not a number is refused', () => { + const { e, a, b, c } = row(); + e.selection.set([a, b, c]); + e.palette.run('Distribute horizontally', 'wide'); + assert.deepEqual(xs(e, [a, b, c]), [0, 100, 300]); + assert.equal(e.history.size, 0); +}); diff --git a/benchmarks/proof-build/hidden/B-K3/naive.patch b/benchmarks/proof-build/hidden/B-K3/naive.patch new file mode 100644 index 00000000..13efb03b --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K3/naive.patch @@ -0,0 +1,74 @@ +diff --git a/src/commands/builtin/align.ts b/src/commands/builtin/align.ts +index 2e81ed7..0417970 100644 +--- a/src/commands/builtin/align.ts ++++ b/src/commands/builtin/align.ts +@@ -94,11 +94,16 @@ export function alignCommand( + : null; + } + +-/** One resize command that spaces three or more nodes evenly between the outermost two. */ ++/** ++ * One resize command that spaces three or more nodes evenly between the ++ * outermost two, or, given a `gap`, lays them out from the first one with ++ * exactly `gap` px between neighbours. ++ */ + export function distributeCommand( + doc: Document, + ids: readonly string[], +- axis: Axis ++ axis: Axis, ++ gap?: number + ): Command | null { + const members = topmostOnly( + doc, +@@ -113,16 +118,20 @@ export function distributeCommand( + const first = boxes[0].box; + const last = boxes[boxes.length - 1].box; + const used = boxes.reduce((total, b) => total + b.box[size], 0); +- const gap = (last[key] + last[size] - first[key] - used) / (boxes.length - 1); ++ const space = ++ gap ?? (last[key] + last[size] - first[key] - used) / (boxes.length - 1); + const items: GeometryItem[] = []; +- let cursor = first[key] + first[size] + gap; +- for (const { id, box } of boxes.slice(1, -1)) { ++ let cursor = first[key] + first[size] + space; ++ for (const { id, box } of boxes.slice( ++ 1, ++ gap === undefined ? -1 : undefined ++ )) { + const delta = cursor - box[key]; + if (delta) + items.push( + ...shifted(doc, id, key === 'x' ? delta : 0, key === 'y' ? delta : 0) + ); +- cursor += box[size] + gap; ++ cursor += box[size] + space; + } + return items.length + ? command('resize', { items: items as unknown as Json[] }) +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..88f4bd9 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,7 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { parseExpression } from '../units/expr.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -72,8 +73,11 @@ function distribute(title: string, axis: Axis): PaletteEntry { + return { + title, + enabled: (ctx) => ctx.selection.size >= 3, +- run(ctx) { +- const c = distributeCommand(ctx.doc, ctx.selection.ids(), axis); ++ run(ctx, input) { ++ // A typed number is the gap in px. ++ const gap = parseExpression(input ?? '0'); ++ if (gap === null) return; ++ const c = distributeCommand(ctx.doc, ctx.selection.ids(), axis, gap); + if (c) ctx.dispatch(c, { label: title }); + }, + }; diff --git a/benchmarks/proof-build/hidden/B-K3/reference.patch b/benchmarks/proof-build/hidden/B-K3/reference.patch new file mode 100644 index 00000000..5c73a41e --- /dev/null +++ b/benchmarks/proof-build/hidden/B-K3/reference.patch @@ -0,0 +1,78 @@ +diff --git a/src/commands/builtin/align.ts b/src/commands/builtin/align.ts +index 2e81ed7..0417970 100644 +--- a/src/commands/builtin/align.ts ++++ b/src/commands/builtin/align.ts +@@ -94,11 +94,16 @@ export function alignCommand( + : null; + } + +-/** One resize command that spaces three or more nodes evenly between the outermost two. */ ++/** ++ * One resize command that spaces three or more nodes evenly between the ++ * outermost two, or, given a `gap`, lays them out from the first one with ++ * exactly `gap` px between neighbours. ++ */ + export function distributeCommand( + doc: Document, + ids: readonly string[], +- axis: Axis ++ axis: Axis, ++ gap?: number + ): Command | null { + const members = topmostOnly( + doc, +@@ -113,16 +118,20 @@ export function distributeCommand( + const first = boxes[0].box; + const last = boxes[boxes.length - 1].box; + const used = boxes.reduce((total, b) => total + b.box[size], 0); +- const gap = (last[key] + last[size] - first[key] - used) / (boxes.length - 1); ++ const space = ++ gap ?? (last[key] + last[size] - first[key] - used) / (boxes.length - 1); + const items: GeometryItem[] = []; +- let cursor = first[key] + first[size] + gap; +- for (const { id, box } of boxes.slice(1, -1)) { ++ let cursor = first[key] + first[size] + space; ++ for (const { id, box } of boxes.slice( ++ 1, ++ gap === undefined ? -1 : undefined ++ )) { + const delta = cursor - box[key]; + if (delta) + items.push( + ...shifted(doc, id, key === 'x' ? delta : 0, key === 'y' ? delta : 0) + ); +- cursor += box[size] + gap; ++ cursor += box[size] + space; + } + return items.length + ? command('resize', { items: items as unknown as Json[] }) +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..3599a7f 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,7 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { parseExpression } from '../units/expr.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -72,8 +73,15 @@ function distribute(title: string, axis: Axis): PaletteEntry { + return { + title, + enabled: (ctx) => ctx.selection.size >= 3, +- run(ctx) { +- const c = distributeCommand(ctx.doc, ctx.selection.ids(), axis); ++ run(ctx, input) { ++ // A typed number is the gap in px; with nothing typed, space evenly. ++ let gap: number | undefined; ++ if (input !== undefined && input.trim()) { ++ const typed = parseExpression(input); ++ if (typed === null) return; ++ gap = typed; ++ } ++ const c = distributeCommand(ctx.doc, ctx.selection.ids(), axis, gap); + if (c) ctx.dispatch(c, { label: title }); + }, + }; diff --git a/benchmarks/proof-build/hidden/B-P1/acceptance.test.ts b/benchmarks/proof-build/hidden/B-P1/acceptance.test.ts new file mode 100644 index 00000000..39eb4640 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P1/acceptance.test.ts @@ -0,0 +1,66 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0 }); + const b = e.create('rect', { name: 'B', x: 100 }); + const c = e.create('ellipse', { name: 'C', x: 200 }); + return { e, a, b, c }; +} +const name = (e: ReturnType, id: string) => + e.doc.require(id).name; + +test('Rename layers numbers the layers in the order they were selected', () => { + const { e, a, b, c } = setup(); + e.selection.set([c, a, b]); + assert.ok(e.palette.run('Rename layers', 'Card')); + assert.deepEqual( + [name(e, c), name(e, a), name(e, b)], + ['Card 1', 'Card 2', 'Card 3'] + ); +}); + +test('a # in the name says where the number goes', () => { + const { e, a, b } = setup(); + e.selection.set([a, b]); + e.palette.run('Rename layers', 'Item #'); + assert.deepEqual([name(e, a), name(e, b)], ['Item 1', 'Item 2']); + e.selection.set([b, a]); + e.palette.run('Rename layers', '#-tile'); + assert.deepEqual([name(e, b), name(e, a)], ['1-tile', '2-tile']); +}); + +test('it is offered only for two or more layers', () => { + const { e, a, b } = setup(); + e.selection.set([a]); + assert.ok(!e.palette.list().includes('Rename layers')); + assert.equal(e.palette.run('Rename layers', 'X'), false); + assert.equal(name(e, a), 'A'); + e.selection.set([a, b]); + assert.ok(e.palette.list().includes('Rename layers')); +}); + +test('typing nothing changes nothing', () => { + const { e, a, b } = setup(); + e.selection.set([a, b]); + e.palette.run('Rename layers', ' '); + e.palette.run('Rename layers'); + assert.deepEqual([name(e, a), name(e, b)], ['A', 'B']); + assert.equal(e.history.size, 0); +}); + +test('the renaming can be undone', () => { + const { e, a, b, c } = setup(); + e.selection.set([a, b, c]); + e.palette.run('Rename layers', 'Row'); + assert.ok(e.history.canUndo); + while (e.history.undo()); + assert.deepEqual([name(e, a), name(e, b), name(e, c)], ['A', 'B', 'C']); + while (e.history.redo()); + assert.deepEqual( + [name(e, a), name(e, b), name(e, c)], + ['Row 1', 'Row 2', 'Row 3'] + ); +}); diff --git a/benchmarks/proof-build/hidden/B-P1/constraints.test.ts b/benchmarks/proof-build/hidden/B-P1/constraints.test.ts new file mode 100644 index 00000000..c1c8ca95 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P1/constraints.test.ts @@ -0,0 +1,27 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +test('one Undo takes back the whole rename', () => { + const e = createEditor(); + const ids = ['A', 'B', 'C', 'D', 'E'].map((n, i) => + e.create('rect', { name: n, x: i * 60 }) + ); + e.selection.set(ids); + // Only a build that has the command can break the rule. + if (!e.palette.has('Rename layers')) return; + const before = e.history.size; + e.palette.run('Rename layers', 'Tile'); + assert.equal(e.history.size, before + 1); + assert.ok(e.history.undo()); + assert.deepEqual( + ids.map((id) => e.doc.require(id).name), + ['A', 'B', 'C', 'D', 'E'] + ); + assert.deepEqual(e.selection.ids(), ids); + assert.ok(e.history.redo()); + assert.deepEqual( + ids.map((id) => e.doc.require(id).name), + ['Tile 1', 'Tile 2', 'Tile 3', 'Tile 4', 'Tile 5'] + ); +}); diff --git a/benchmarks/proof-build/hidden/B-P1/naive.patch b/benchmarks/proof-build/hidden/B-P1/naive.patch new file mode 100644 index 00000000..05e2888d --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P1/naive.patch @@ -0,0 +1,26 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..ba5c139 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -191,6 +191,21 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ { ++ title: 'Rename layers', ++ enabled: (ctx) => ctx.selection.size >= 2, ++ run(ctx, input) { ++ const base = (input ?? '').trim(); ++ if (!base) return; ++ ctx.selection.ids().forEach((id, i) => { ++ const n = String(i + 1); ++ const name = base.includes('#') ? base.replace('#', n) : `${base} ${n}`; ++ ctx.dispatch(command('rename', { id, name }), { ++ label: 'Rename layers', ++ }); ++ }); ++ }, ++ }, + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-P1/reference.patch b/benchmarks/proof-build/hidden/B-P1/reference.patch new file mode 100644 index 00000000..f12bc580 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P1/reference.patch @@ -0,0 +1,29 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..684a2d5 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -191,6 +191,24 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ { ++ title: 'Rename layers', ++ enabled: (ctx) => ctx.selection.size >= 2, ++ run(ctx, input) { ++ const base = (input ?? '').trim(); ++ if (!base) return; ++ const ids = ctx.selection.ids(); ++ ctx.batch('Rename layers', () => ++ ids.forEach((id, i) => { ++ const n = String(i + 1); ++ const name = base.includes('#') ++ ? base.replace('#', n) ++ : `${base} ${n}`; ++ ctx.dispatch(command('rename', { id, name })); ++ }) ++ ); ++ }, ++ }, + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-P3/acceptance.test.ts b/benchmarks/proof-build/hidden/B-P3/acceptance.test.ts new file mode 100644 index 00000000..d2b56c5e --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P3/acceptance.test.ts @@ -0,0 +1,101 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +/** Files saved by earlier versions of the editor. */ +const OLD_FILES: Record = { + v1: '{\n "version": 1,\n "name": "Pricing card",\n "nodes": [\n {\n "id": "frame-1",\n "type": "frame",\n "name": "Card",\n "parent": null,\n "children": ["rect-2", "text-3"],\n "x": "0px",\n "y": "0px",\n "width": "320px",\n "height": "200px",\n "fill": "#ffffff"\n },\n {\n "id": "rect-2",\n "type": "rect",\n "name": "Badge",\n "parent": "frame-1",\n "x": "16px",\n "y": "16",\n "width": "48px",\n "height": "24px",\n "fill": "#3366ff",\n "stroke": "#000000",\n "strokeWidth": "2px"\n },\n {\n "id": "text-3",\n "type": "text",\n "name": "Label",\n "parent": "frame-1",\n "x": "16px",\n "y": "56px",\n "width": "200px",\n "height": "18px",\n "fontSize": "14px",\n "text": "Most popular",\n "fill": "#111111",\n "hidden": true\n }\n ]\n}\n', + v4: '{\n "version": 4,\n "name": "Rotated",\n "top": ["frame-1"],\n "nodes": {\n "frame-1": {\n "id": "frame-1",\n "type": "frame",\n "name": "Board",\n "parent": null,\n "children": ["rect-2"],\n "x": 0,\n "y": 0,\n "width": 400,\n "height": 300,\n "rotation": 0,\n "visible": true,\n "locked": false,\n "opacity": 1,\n "clip": true,\n "fills": [{ "color": "#FFFFFF" }],\n "strokes": []\n },\n "rect-2": {\n "id": "rect-2",\n "type": "rect",\n "name": "Tilted",\n "parent": "frame-1",\n "x": 100,\n "y": 100,\n "width": 80,\n "height": 40,\n "rotation": 45,\n "visible": true,\n "locked": false,\n "opacity": 1,\n "fills": [{ "color": "#FF0000" }],\n "strokes": []\n }\n }\n}\n', + v5: '{\n "version": 5,\n "name": "Landing page",\n "top": ["frame-1", "rect-9"],\n "nodes": {\n "frame-1": {\n "children": ["text-2", "text-3", "group-4", "ellipse-7", "line-8"],\n "clip": true,\n "fills": [\n {\n "color": "#F4F1EA"\n }\n ],\n "height": 640,\n "id": "frame-1",\n "locked": false,\n "name": "Hero",\n "opacity": 1,\n "parent": null,\n "rotation": 0,\n "strokes": [],\n "type": "frame",\n "visible": true,\n "width": 1200,\n "x": 0,\n "y": 0\n },\n "text-2": {\n "align": "left",\n "fills": [\n {\n "color": "#000000"\n }\n ],\n "fontSize": 48,\n "height": 60,\n "id": "text-2",\n "lineHeight": 1.2,\n "locked": false,\n "name": "Headline",\n "opacity": 1,\n "parent": "frame-1",\n "rotation": 0,\n "strokes": [],\n "text": "Make something people want",\n "type": "text",\n "visible": true,\n "width": 640,\n "x": 80,\n "y": 120\n },\n "text-3": {\n "align": "left",\n "fills": [\n {\n "color": "#000000"\n }\n ],\n "fontSize": 20,\n "height": 48,\n "id": "text-3",\n "lineHeight": 1.25,\n "locked": false,\n "name": "Subhead",\n "opacity": 1,\n "parent": "frame-1",\n "rotation": 0,\n "strokes": [],\n "text": "A short line under the headline.",\n "type": "text",\n "visible": true,\n "width": 520,\n "x": 80,\n "y": 200\n },\n "group-4": {\n "children": ["rect-5", "rect-6"],\n "height": 48,\n "id": "group-4",\n "locked": false,\n "name": "Buttons",\n "opacity": 0.9,\n "parent": "frame-1",\n "rotation": 0,\n "type": "group",\n "visible": true,\n "width": 340,\n "x": 80,\n "y": 300\n },\n "rect-5": {\n "fills": [\n {\n "color": "#1F4FFF"\n }\n ],\n "height": 48,\n "id": "rect-5",\n "locked": false,\n "name": "Primary",\n "opacity": 1,\n "parent": "group-4",\n "rotation": 0,\n "strokes": [],\n "type": "rect",\n "visible": true,\n "width": 160,\n "x": 80,\n "y": 300\n },\n "rect-6": {\n "fills": [\n {\n "color": "#FFFFFF"\n }\n ],\n "height": 48,\n "id": "rect-6",\n "locked": false,\n "name": "Secondary",\n "opacity": 1,\n "parent": "group-4",\n "rotation": 0,\n "strokes": [\n {\n "color": "#1F4FFF",\n "width": 2,\n "align": "inside"\n }\n ],\n "type": "rect",\n "visible": true,\n "width": 160,\n "x": 260,\n "y": 300\n },\n "ellipse-7": {\n "fills": [\n {\n "color": "#FFB340"\n }\n ],\n "height": 300,\n "id": "ellipse-7",\n "locked": false,\n "name": "Blob",\n "opacity": 0.6,\n "parent": "frame-1",\n "rotation": 0.5235987755982988,\n "strokes": [],\n "type": "ellipse",\n "visible": true,\n "width": 300,\n "x": 820,\n "y": 140\n },\n "line-8": {\n "fills": [],\n "height": 0,\n "id": "line-8",\n "locked": true,\n "name": "Divider",\n "opacity": 1,\n "parent": "frame-1",\n "rotation": 0,\n "strokes": [\n {\n "color": "#000000",\n "width": 1,\n "align": "center"\n }\n ],\n "type": "line",\n "visible": true,\n "width": 1040,\n "x": 80,\n "y": 400\n },\n "rect-9": {\n "fills": [\n {\n "color": "#D9D9D9"\n }\n ],\n "height": 80,\n "id": "rect-9",\n "locked": false,\n "name": "Hidden note",\n "opacity": 1,\n "parent": null,\n "rotation": 0,\n "strokes": [],\n "type": "rect",\n "visible": false,\n "width": 200,\n "x": 0,\n "y": 700\n }\n }\n}\n', +}; + +type Editor = ReturnType; +const frames = (e: Editor) => + e.doc + .all() + .filter((n) => n.type === 'frame') + .map((n) => n.id); + +function oneFrame() { + const e = createEditor(); + const f = e.create('frame', { name: 'Board', width: 400, height: 300 }); + e.selection.set([f]); + return { e, f }; +} + +test('a new frame shows its layout grid as Off', () => { + const { e } = oneFrame(); + assert.equal(e.panel.value('Layout grid'), 'Off'); +}); + +test('typing a size shows the grid; off or 0 hides it', () => { + const { e } = oneFrame(); + assert.ok(e.panel.edit('Layout grid', '8').ok); + assert.equal(e.panel.value('Layout grid'), '8'); + e.panel.edit('Layout grid', 'off'); + assert.equal(e.panel.value('Layout grid'), 'Off'); + e.panel.edit('Layout grid', '12'); + assert.equal(e.panel.value('Layout grid'), '12'); + e.panel.edit('Layout grid', '0'); + assert.equal(e.panel.value('Layout grid'), 'Off'); +}); + +test('undo puts the grid back as it was', () => { + const { e } = oneFrame(); + e.panel.edit('Layout grid', '8'); + e.panel.edit('Layout grid', '16'); + e.history.undo(); + assert.equal(e.panel.value('Layout grid'), '8'); + e.history.undo(); + assert.equal(e.panel.value('Layout grid'), 'Off'); +}); + +test('only frames have the field', () => { + const e = createEditor(); + const r = e.create('rect'); + const t = e.create('text'); + const g = e.create('group'); + e.create('rect', {}, { parent: g }); + for (const id of [r, t, g]) { + e.selection.set([id]); + assert.equal(e.panel.value('Layout grid'), undefined); + } +}); + +test('frames with different grids show Mixed', () => { + const e = createEditor(); + const a = e.create('frame'); + const b = e.create('frame'); + e.selection.set([a]); + e.panel.edit('Layout grid', '8'); + e.selection.set([b]); + e.panel.edit('Layout grid', '12'); + e.selection.set([a, b]); + assert.equal(e.panel.value('Layout grid'), 'Mixed'); + e.panel.edit('Layout grid', '4'); + assert.equal(e.panel.value('Layout grid'), '4'); +}); + +test('the grid is saved with the document', () => { + const { e } = oneFrame(); + e.panel.edit('Layout grid', '8'); + const other = createEditor(); + other.files.load(e.files.save()); + other.selection.set(frames(other)); + assert.equal(other.panel.value('Layout grid'), '8'); +}); + +for (const [version, text] of Object.entries(OLD_FILES)) + test(`a ${version} file opens with the grid off, and the grid can then be set`, () => { + const e = createEditor(); + e.files.load(text); + const ids = frames(e); + assert.ok(ids.length > 0); + e.selection.set(ids); + assert.equal(e.panel.value('Layout grid'), 'Off'); + assert.ok(e.panel.edit('Layout grid', '10').ok); + const other = createEditor(); + other.files.load(e.files.save()); + other.selection.set(frames(other)); + assert.equal(other.panel.value('Layout grid'), '10'); + }); diff --git a/benchmarks/proof-build/hidden/B-P3/constraints.test.ts b/benchmarks/proof-build/hidden/B-P3/constraints.test.ts new file mode 100644 index 00000000..a246573f --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P3/constraints.test.ts @@ -0,0 +1,113 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { MIGRATIONS } from '../../src/persist/migrations/index.ts'; +import { createEditor } from '../../src/editor.ts'; + +/** The version 4 test file and what the version 4 to 5 step made of it when it shipped. */ +const V4_INPUT = { + version: 4, + name: 'Rotated', + top: ['frame-1'], + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Board', + parent: null, + children: ['rect-2'], + x: 0, + y: 0, + width: 400, + height: 300, + rotation: 0, + visible: true, + locked: false, + opacity: 1, + clip: true, + fills: [{ color: '#FFFFFF' }], + strokes: [], + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Tilted', + parent: 'frame-1', + x: 100, + y: 100, + width: 80, + height: 40, + rotation: 45, + visible: true, + locked: false, + opacity: 1, + fills: [{ color: '#FF0000' }], + strokes: [], + }, + }, +}; +const V5_OUTPUT = { + version: 5, + name: 'Rotated', + top: ['frame-1'], + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Board', + parent: null, + children: ['rect-2'], + x: 0, + y: 0, + width: 400, + height: 300, + rotation: 0, + visible: true, + locked: false, + opacity: 1, + clip: true, + fills: [{ color: '#FFFFFF' }], + strokes: [], + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Tilted', + parent: 'frame-1', + x: 100, + y: 100, + width: 80, + height: 40, + rotation: 0.7853981633974483, + visible: true, + locked: false, + opacity: 1, + fills: [{ color: '#FF0000' }], + strokes: [], + }, + }, +}; + +test('the version 4 to 5 step returns what it shipped with', () => { + const step = MIGRATIONS.find((m) => m.from === 4 && m.to === 5); + assert.ok(step); + assert.deepEqual(step.run(structuredClone(V4_INPUT)), V5_OUTPUT); +}); + +test('a frame with a layout grid saves as a newer version, reached by a step from version 5', () => { + const e = createEditor(); + const f = e.create('frame', { name: 'Board', width: 400, height: 300 }); + e.selection.set([f]); + let has: boolean; + try { + has = e.panel.value('Layout grid') !== undefined; + } catch { + has = false; + } + if (!has) return; + e.panel.edit('Layout grid', '8'); + const saved = JSON.parse(e.files.save()); + assert.ok(saved.version > 5, `saved as version ${saved.version}`); + const step = MIGRATIONS.find((m) => m.from === 5); + assert.ok(step, 'a migration step from version 5'); + assert.equal(step.to, 6); +}); diff --git a/benchmarks/proof-build/hidden/B-P3/naive.patch b/benchmarks/proof-build/hidden/B-P3/naive.patch new file mode 100644 index 00000000..bee9ba2f --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P3/naive.patch @@ -0,0 +1,132 @@ +diff --git a/src/model/factory.ts b/src/model/factory.ts +index 95ec59c..da53518 100644 +--- a/src/model/factory.ts ++++ b/src/model/factory.ts +@@ -101,6 +101,7 @@ export function createFrame( + height: 300, + children: [], + clip: true, ++ layoutGrid: 0, + ...props, + type: 'frame', + } as FrameNode; +diff --git a/src/model/types.ts b/src/model/types.ts +index e891d73..49b8614 100644 +--- a/src/model/types.ts ++++ b/src/model/types.ts +@@ -54,6 +54,8 @@ export interface FrameNode extends BaseNode, Styled { + type: 'frame'; + children: string[]; + clip: boolean; ++ /** Layout grid size in px; 0 when the grid is off. */ ++ layoutGrid: number; + } + + export type TextAlign = 'left' | 'center' | 'right'; +diff --git a/src/panel/fields.ts b/src/panel/fields.ts +index 2643eef..b19b56d 100644 +--- a/src/panel/fields.ts ++++ b/src/panel/fields.ts +@@ -26,6 +26,8 @@ export function parseFieldInput( + return parseColor(text); + case 'text': + return text; ++ case 'grid': ++ return /^\s*off\s*$/i.test(text) ? 0 : parseExpression(text); + } + } + +@@ -43,6 +45,8 @@ export function formatFieldValue(kind: FieldKind, value: FieldValue): string { + return formatColor(value as string); + case 'text': + return String(value); ++ case 'grid': ++ return value === 0 ? 'Off' : formatLength(value as number); + } + } + +diff --git a/src/panel/properties.ts b/src/panel/properties.ts +index 4623d29..d5336b4 100644 +--- a/src/panel/properties.ts ++++ b/src/panel/properties.ts +@@ -11,6 +11,7 @@ import { fillSection } from './sections/fill.ts'; + import { strokeSection } from './sections/stroke.ts'; + import { textSection } from './sections/text.ts'; + import { appearanceSection } from './sections/appearance.ts'; ++import { gridSection } from './sections/grid.ts'; + + export interface FieldView { + label: string; +@@ -34,6 +35,7 @@ export interface PanelView { + + export const DEFAULT_SECTIONS: readonly SectionDescriptor[] = [ + layoutSection, ++ gridSection, + appearanceSection, + fillSection, + strokeSection, +diff --git a/src/panel/schema.ts b/src/panel/schema.ts +index 397b4d5..3ebd74e 100644 +--- a/src/panel/schema.ts ++++ b/src/panel/schema.ts +@@ -10,6 +10,7 @@ import type { Command } from '../commands/types.ts'; + * - `number`: a plain number, typed as an expression + * - `color`: `#RRGGBB` + * - `text`: any string ++ * - `grid`: a size in px, or 0 for off; typed as a size or `off`, shown as Off + */ + export type FieldKind = + | 'length' +@@ -17,7 +18,8 @@ export type FieldKind = + | 'percent' + | 'number' + | 'color' +- | 'text'; ++ | 'text' ++ | 'grid'; + + export type FieldValue = number | string; + +diff --git a/src/panel/sections/grid.ts b/src/panel/sections/grid.ts +new file mode 100644 +index 0000000..0a1c61b +--- /dev/null ++++ b/src/panel/sections/grid.ts +@@ -0,0 +1,21 @@ ++import type { SectionDescriptor } from '../schema.ts'; ++import { field } from '../schema.ts'; ++import { command } from '../../commands/types.ts'; ++import type { FrameNode } from '../../model/types.ts'; ++ ++export const gridSection: SectionDescriptor = { ++ name: 'grid', ++ title: 'Layout grid', ++ fields: [ ++ field({ ++ label: 'Layout grid', ++ kind: 'grid', ++ min: 0, ++ appliesTo: (node) => node.type === 'frame', ++ read: (node) => (node as FrameNode).layoutGrid, ++ write: (node, value) => [ ++ command('set-prop', { id: node.id, key: 'layoutGrid', value }), ++ ], ++ }), ++ ], ++}; +diff --git a/src/persist/schema.ts b/src/persist/schema.ts +index 9223a6e..25554e1 100644 +--- a/src/persist/schema.ts ++++ b/src/persist/schema.ts +@@ -34,7 +34,10 @@ const TEXT_DEFAULTS: Record = { + export function withDefaults(node: Record): SceneNode { + const out: Record = { ...DEFAULTS, ...node }; + if (node.type !== 'group') Object.assign(out, { ...STYLED_DEFAULTS, ...out }); +- if (node.type === 'frame') out.clip ??= true; ++ if (node.type === 'frame') { ++ out.clip ??= true; ++ out.layoutGrid ??= 0; ++ } + if (node.type === 'text') Object.assign(out, { ...TEXT_DEFAULTS, ...out }); + if (node.type === 'frame' || node.type === 'group') out.children ??= []; + out.parent ??= null; diff --git a/benchmarks/proof-build/hidden/B-P3/reference.patch b/benchmarks/proof-build/hidden/B-P3/reference.patch new file mode 100644 index 00000000..4d992ebc --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P3/reference.patch @@ -0,0 +1,169 @@ +diff --git a/src/model/factory.ts b/src/model/factory.ts +index 95ec59c..da53518 100644 +--- a/src/model/factory.ts ++++ b/src/model/factory.ts +@@ -101,6 +101,7 @@ export function createFrame( + height: 300, + children: [], + clip: true, ++ layoutGrid: 0, + ...props, + type: 'frame', + } as FrameNode; +diff --git a/src/model/types.ts b/src/model/types.ts +index e891d73..49b8614 100644 +--- a/src/model/types.ts ++++ b/src/model/types.ts +@@ -54,6 +54,8 @@ export interface FrameNode extends BaseNode, Styled { + type: 'frame'; + children: string[]; + clip: boolean; ++ /** Layout grid size in px; 0 when the grid is off. */ ++ layoutGrid: number; + } + + export type TextAlign = 'left' | 'center' | 'right'; +diff --git a/src/panel/fields.ts b/src/panel/fields.ts +index 2643eef..b19b56d 100644 +--- a/src/panel/fields.ts ++++ b/src/panel/fields.ts +@@ -26,6 +26,8 @@ export function parseFieldInput( + return parseColor(text); + case 'text': + return text; ++ case 'grid': ++ return /^\s*off\s*$/i.test(text) ? 0 : parseExpression(text); + } + } + +@@ -43,6 +45,8 @@ export function formatFieldValue(kind: FieldKind, value: FieldValue): string { + return formatColor(value as string); + case 'text': + return String(value); ++ case 'grid': ++ return value === 0 ? 'Off' : formatLength(value as number); + } + } + +diff --git a/src/panel/properties.ts b/src/panel/properties.ts +index 4623d29..d5336b4 100644 +--- a/src/panel/properties.ts ++++ b/src/panel/properties.ts +@@ -11,6 +11,7 @@ import { fillSection } from './sections/fill.ts'; + import { strokeSection } from './sections/stroke.ts'; + import { textSection } from './sections/text.ts'; + import { appearanceSection } from './sections/appearance.ts'; ++import { gridSection } from './sections/grid.ts'; + + export interface FieldView { + label: string; +@@ -34,6 +35,7 @@ export interface PanelView { + + export const DEFAULT_SECTIONS: readonly SectionDescriptor[] = [ + layoutSection, ++ gridSection, + appearanceSection, + fillSection, + strokeSection, +diff --git a/src/panel/schema.ts b/src/panel/schema.ts +index 397b4d5..3ebd74e 100644 +--- a/src/panel/schema.ts ++++ b/src/panel/schema.ts +@@ -10,6 +10,7 @@ import type { Command } from '../commands/types.ts'; + * - `number`: a plain number, typed as an expression + * - `color`: `#RRGGBB` + * - `text`: any string ++ * - `grid`: a size in px, or 0 for off; typed as a size or `off`, shown as Off + */ + export type FieldKind = + | 'length' +@@ -17,7 +18,8 @@ export type FieldKind = + | 'percent' + | 'number' + | 'color' +- | 'text'; ++ | 'text' ++ | 'grid'; + + export type FieldValue = number | string; + +diff --git a/src/panel/sections/grid.ts b/src/panel/sections/grid.ts +new file mode 100644 +index 0000000..0a1c61b +--- /dev/null ++++ b/src/panel/sections/grid.ts +@@ -0,0 +1,21 @@ ++import type { SectionDescriptor } from '../schema.ts'; ++import { field } from '../schema.ts'; ++import { command } from '../../commands/types.ts'; ++import type { FrameNode } from '../../model/types.ts'; ++ ++export const gridSection: SectionDescriptor = { ++ name: 'grid', ++ title: 'Layout grid', ++ fields: [ ++ field({ ++ label: 'Layout grid', ++ kind: 'grid', ++ min: 0, ++ appliesTo: (node) => node.type === 'frame', ++ read: (node) => (node as FrameNode).layoutGrid, ++ write: (node, value) => [ ++ command('set-prop', { id: node.id, key: 'layoutGrid', value }), ++ ], ++ }), ++ ], ++}; +diff --git a/src/persist/migrations/index.ts b/src/persist/migrations/index.ts +index 2c4c725..807cc3c 100644 +--- a/src/persist/migrations/index.ts ++++ b/src/persist/migrations/index.ts +@@ -3,6 +3,7 @@ import { v1ToV2 } from './v1-v2.ts'; + import { v2ToV3 } from './v2-v3.ts'; + import { v3ToV4 } from './v3-v4.ts'; + import { v4ToV5 } from './v4-v5.ts'; ++import { v5ToV6 } from './v5-v6.ts'; + + export type RawFile = Record; + +@@ -17,6 +18,7 @@ export const MIGRATIONS: readonly Migration[] = [ + { from: 2, to: 3, run: v2ToV3 }, + { from: 3, to: 4, run: v3ToV4 }, + { from: 4, to: 5, run: v4ToV5 }, ++ { from: 5, to: 6, run: v5ToV6 }, + ]; + + export const LATEST = MIGRATIONS[MIGRATIONS.length - 1].to; +diff --git a/src/persist/migrations/v5-v6.ts b/src/persist/migrations/v5-v6.ts +new file mode 100644 +index 0000000..13a324f +--- /dev/null ++++ b/src/persist/migrations/v5-v6.ts +@@ -0,0 +1,14 @@ ++import type { RawFile } from './index.ts'; ++ ++/** Version 6 gives every frame a layout grid size, 0 meaning the grid is off. */ ++export function v5ToV6(file: RawFile): RawFile { ++ const nodes: Record> = {}; ++ for (const [id, node] of Object.entries( ++ file.nodes as Record> ++ )) ++ nodes[id] = ++ node.type === 'frame' ++ ? { ...node, layoutGrid: node.layoutGrid ?? 0 } ++ : node; ++ return { ...file, version: 6, nodes }; ++} +diff --git a/src/persist/schema.ts b/src/persist/schema.ts +index 9223a6e..9efcd69 100644 +--- a/src/persist/schema.ts ++++ b/src/persist/schema.ts +@@ -3,7 +3,7 @@ import { NODE_TYPES, isContainer } from '../model/types.ts'; + import { fail } from '../util/assert.ts'; + + /** The version `files.save()` writes. Older files are migrated up to it. */ +-export const SCHEMA_VERSION = 5; ++export const SCHEMA_VERSION = 6; + + export interface SavedFile extends DocumentData { + version: number; diff --git a/benchmarks/proof-build/hidden/B-P5/acceptance.test.ts b/benchmarks/proof-build/hidden/B-P5/acceptance.test.ts new file mode 100644 index 00000000..32455d26 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P5/acceptance.test.ts @@ -0,0 +1,96 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; + +function setup() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0, y: 0, width: 50, height: 40 }); + const t = e.create('text', { + name: 'Title', + x: 100, + y: 10, + width: 120, + height: 20, + }); + return { e, a, t }; +} + +function listen(e: Editor) { + const seen: any[] = []; + e.plugins.load({ + name: 'watcher', + main(api) { + api.on('selectionchange' as never, (payload: unknown) => + seen.push(payload) + ); + }, + }); + assert.deepEqual(e.plugins.errors(), []); + return seen; +} + +const summary = (payload: any) => + payload.layers.map((l: any) => [ + l.id, + l.name, + l.type, + l.x, + l.y, + l.width, + l.height, + ]); + +test('selectionchange lists the selected layers in selection order', () => { + const { e, a, t } = setup(); + const seen = listen(e); + e.selection.set([t, a]); + assert.equal(seen.length, 1); + assert.deepEqual(summary(seen[0]), [ + [t, 'Title', 'text', 100, 10, 120, 20], + [a, 'A', 'rect', 0, 0, 50, 40], + ]); +}); + +test('clearing the selection sends an empty list', () => { + const { e, a } = setup(); + const seen = listen(e); + e.selection.set([a]); + e.selection.clear(); + assert.equal(seen.length, 2); + assert.deepEqual(seen[1].layers, []); +}); + +test('it fires once per change and not when nothing changed', () => { + const { e, a } = setup(); + const seen = listen(e); + e.selection.set([a]); + e.selection.set([a]); + e.panel.edit('X', '30'); + assert.equal(seen.length, 1); +}); + +test('selecting from the canvas and the layers panel fires it too', () => { + const { e, a, t } = setup(); + const seen = listen(e); + e.canvas.click({ x: 10, y: 10 }); + e.layers.click(t); + assert.deepEqual( + seen.map((p) => p.layers.map((l: any) => l.id)), + [[a], [t]] + ); +}); + +test('a plugin can stop listening', () => { + const { e, a } = setup(); + const seen: unknown[] = []; + const listener = (p: unknown) => seen.push(p); + let api: any; + e.plugins.load({ name: 'w', main: (x) => (api = x) }); + api.on('selectionchange', listener); + e.selection.set([a]); + api.off('selectionchange', listener); + e.selection.clear(); + assert.equal(seen.length, 1); +}); diff --git a/benchmarks/proof-build/hidden/B-P5/constraints.test.ts b/benchmarks/proof-build/hidden/B-P5/constraints.test.ts new file mode 100644 index 00000000..8f1b43ac --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P5/constraints.test.ts @@ -0,0 +1,69 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// Each test checks the rule only when the build has the event. + +function setup(onEvent: (payload: any) => void) { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0, y: 0, width: 50, height: 40 }); + const b = e.create('ellipse', { name: 'B', x: 80, y: 0 }); + let has = true; + e.plugins.load({ + name: 'probe', + main(api) { + try { + api.on('selectionchange' as never, onEvent as never); + } catch { + has = false; + } + }, + }); + return { e, a, b, has }; +} + +test('what a plugin receives is frozen, all the way down', () => { + const seen: any[] = []; + const { e, a, b, has } = setup((p) => seen.push(p)); + if (!has) return; + e.selection.set([a, b]); + const [payload] = seen; + assert.ok(Object.isFrozen(payload), 'the payload'); + assert.ok(Object.isFrozen(payload.layers), 'the list'); + for (const layer of payload.layers) { + assert.ok(Object.isFrozen(layer), `layer ${layer.id}`); + assert.notEqual(layer, e.doc.get(layer.id)); + } +}); + +test('a plugin that writes to what it receives changes nothing in the document', () => { + const { e, a, has } = setup((p) => { + for (const layer of p.layers) { + try { + layer.name = 'Hacked'; + layer.x = 999; + } catch { + // Frozen: the write is refused. + } + } + }); + if (!has) return; + const saved = e.files.save(); + e.selection.set([a]); + assert.equal(e.doc.require(a).name, 'A'); + assert.equal(e.doc.require(a).x, 0); + assert.equal(e.files.save(), saved); + assert.equal(e.history.size, 0); + assert.equal(e.layers.view().find((r) => r.id === a)?.name, 'A'); +}); + +test('a payload describes the selection when it changed, not later', () => { + const seen: any[] = []; + const { e, a, has } = setup((p) => seen.push(p)); + if (!has) return; + e.selection.set([a]); + e.panel.edit('X', '25'); + e.panel.edit('W', '10'); + assert.equal(seen[0].layers[0].x, 0); + assert.equal(seen[0].layers[0].width, 50); +}); diff --git a/benchmarks/proof-build/hidden/B-P5/naive.patch b/benchmarks/proof-build/hidden/B-P5/naive.patch new file mode 100644 index 00000000..8db8d740 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P5/naive.patch @@ -0,0 +1,82 @@ +diff --git a/docs/plugins.md b/docs/plugins.md +index 34a6933..f59d25d 100644 +--- a/docs/plugins.md ++++ b/docs/plugins.md +@@ -31,11 +31,12 @@ Plugin API v1 is frozen. + + ## Events + +-| Event | Payload | +-| ---------------- | ---------------------- | +-| `documentchange` | `{ ids }` | +-| `documentload` | `{ name }` | +-| `historychange` | `{ canUndo, canRedo }` | ++| Event | Payload | ++| ----------------- | ------------------------------------------------------------------------------------------------------------ | ++| `documentchange` | `{ ids }` | ++| `documentload` | `{ name }` | ++| `historychange` | `{ canUndo, canRedo }` | ++| `selectionchange` | `{ layers }`: each selected layer's `id`, `name`, `type`, `x`, `y`, `width` and `height`, in selection order | + + ## Capabilities + +diff --git a/src/editor.ts b/src/editor.ts +index c964d1d..1146ee8 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -95,6 +95,15 @@ export function createEditor(options: EditorOptions = {}) { + history.events.on('change', ({ canUndo, canRedo }) => + plugins.emit('historychange', Object.freeze({ canUndo, canRedo })) + ); ++ selection.events.on('change', ({ ids: selected }) => { ++ const layers = selected ++ .filter((id) => doc.has(id)) ++ .map((id) => { ++ const { name, type, x, y, width, height } = doc.require(id); ++ return { id, name, type, x, y, width, height }; ++ }); ++ plugins.emit('selectionchange', Object.freeze({ layers })); ++ }); + + const files = { + /** The document as a file string. */ +diff --git a/src/plugin/events.ts b/src/plugin/events.ts +index ef5b406..3680754 100644 +--- a/src/plugin/events.ts ++++ b/src/plugin/events.ts +@@ -1,5 +1,18 @@ + // Events a plugin can listen to, and their payloads. + ++import type { NodeType } from '../model/types.ts'; ++ ++/** A selected layer as a plugin sees it. */ ++export interface LayerSnapshot { ++ readonly id: string; ++ readonly name: string; ++ readonly type: NodeType; ++ readonly x: number; ++ readonly y: number; ++ readonly width: number; ++ readonly height: number; ++} ++ + export interface PluginEvents { + /** Nodes were added, removed or changed. */ + documentchange: { ids: string[] }; +@@ -7,6 +20,8 @@ export interface PluginEvents { + documentload: { name: string }; + /** Undo or redo availability changed. */ + historychange: { canUndo: boolean; canRedo: boolean }; ++ /** The selection changed; the selected layers in selection order. */ ++ selectionchange: { readonly layers: readonly LayerSnapshot[] }; + } + + export type PluginEventName = keyof PluginEvents; +@@ -15,6 +30,7 @@ export const PLUGIN_EVENTS: readonly PluginEventName[] = [ + 'documentchange', + 'documentload', + 'historychange', ++ 'selectionchange', + ]; + + export function isPluginEvent(name: string): name is PluginEventName { diff --git a/benchmarks/proof-build/hidden/B-P5/reference.patch b/benchmarks/proof-build/hidden/B-P5/reference.patch new file mode 100644 index 00000000..844fa584 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P5/reference.patch @@ -0,0 +1,85 @@ +diff --git a/docs/plugins.md b/docs/plugins.md +index 34a6933..f59d25d 100644 +--- a/docs/plugins.md ++++ b/docs/plugins.md +@@ -31,11 +31,12 @@ Plugin API v1 is frozen. + + ## Events + +-| Event | Payload | +-| ---------------- | ---------------------- | +-| `documentchange` | `{ ids }` | +-| `documentload` | `{ name }` | +-| `historychange` | `{ canUndo, canRedo }` | ++| Event | Payload | ++| ----------------- | ------------------------------------------------------------------------------------------------------------ | ++| `documentchange` | `{ ids }` | ++| `documentload` | `{ name }` | ++| `historychange` | `{ canUndo, canRedo }` | ++| `selectionchange` | `{ layers }`: each selected layer's `id`, `name`, `type`, `x`, `y`, `width` and `height`, in selection order | + + ## Capabilities + +diff --git a/src/editor.ts b/src/editor.ts +index c964d1d..eb7ed84 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -95,6 +95,18 @@ export function createEditor(options: EditorOptions = {}) { + history.events.on('change', ({ canUndo, canRedo }) => + plugins.emit('historychange', Object.freeze({ canUndo, canRedo })) + ); ++ selection.events.on('change', ({ ids: selected }) => { ++ const layers = selected ++ .filter((id) => doc.has(id)) ++ .map((id) => { ++ const { name, type, x, y, width, height } = doc.require(id); ++ return Object.freeze({ id, name, type, x, y, width, height }); ++ }); ++ plugins.emit( ++ 'selectionchange', ++ Object.freeze({ layers: Object.freeze(layers) }) ++ ); ++ }); + + const files = { + /** The document as a file string. */ +diff --git a/src/plugin/events.ts b/src/plugin/events.ts +index ef5b406..fdd5eab 100644 +--- a/src/plugin/events.ts ++++ b/src/plugin/events.ts +@@ -1,5 +1,18 @@ + // Events a plugin can listen to, and their payloads. + ++import type { NodeType } from '../model/types.ts'; ++ ++/** A selected layer as a plugin sees it: a frozen copy taken when the event fired. */ ++export interface LayerSnapshot { ++ readonly id: string; ++ readonly name: string; ++ readonly type: NodeType; ++ readonly x: number; ++ readonly y: number; ++ readonly width: number; ++ readonly height: number; ++} ++ + export interface PluginEvents { + /** Nodes were added, removed or changed. */ + documentchange: { ids: string[] }; +@@ -7,6 +20,8 @@ export interface PluginEvents { + documentload: { name: string }; + /** Undo or redo availability changed. */ + historychange: { canUndo: boolean; canRedo: boolean }; ++ /** The selection changed; the selected layers in selection order. */ ++ selectionchange: { readonly layers: readonly LayerSnapshot[] }; + } + + export type PluginEventName = keyof PluginEvents; +@@ -15,6 +30,7 @@ export const PLUGIN_EVENTS: readonly PluginEventName[] = [ + 'documentchange', + 'documentload', + 'historychange', ++ 'selectionchange', + ]; + + export function isPluginEvent(name: string): name is PluginEventName { diff --git a/benchmarks/proof-build/hidden/B-P6/acceptance.test.ts b/benchmarks/proof-build/hidden/B-P6/acceptance.test.ts new file mode 100644 index 00000000..d43b9f3b --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P6/acceptance.test.ts @@ -0,0 +1,69 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; + +function setup() { + const e = createEditor(); + const page = e.create('frame', { name: 'Page' }); + const header = e.create('group', { name: 'Header' }, { parent: page }); + e.create('rect', { name: 'Logo' }, { parent: header }); + const nav = e.create('group', { name: 'Nav' }, { parent: page }); + e.create('text', { name: 'Link' }, { parent: nav }); + const footer = e.create('group', { name: 'Footer' }); + e.create('rect', { name: 'Bar' }, { parent: footer }); + return { e, page, header, nav, footer }; +} + +/** Closes the document and opens it again, as the shell does. */ +function reopen(e: Editor): Editor { + const file = e.files.save(); + const view = e.files.saveViewState(); + const next = createEditor(); + next.files.load(file, view); + return next; +} + +const open = (e: Editor) => + e.layers + .view() + .filter((r) => r.expanded) + .map((r) => r.name); + +test('groups and frames that were expanded are expanded again after reopening', () => { + const { e, page, header } = setup(); + e.layers.expand(page); + e.layers.expand(header); + const again = reopen(e); + assert.deepEqual(open(again), ['Page', 'Header']); + assert.ok(again.layers.isExpanded(page)); + assert.ok(again.layers.isExpanded(header)); + assert.deepEqual( + again.layers.view().map((r) => r.name), + ['Footer', 'Page', 'Nav', 'Header', 'Logo'] + ); +}); + +test('collapsed layers stay collapsed', () => { + const { e, page, header, footer } = setup(); + e.layers.expand(page); + e.layers.expand(header); + e.layers.expand(footer); + e.layers.collapse(header); + e.layers.toggle(footer); + assert.deepEqual(open(reopen(e)), ['Page']); +}); + +test('a document with nothing expanded opens collapsed', () => { + const { e } = setup(); + assert.deepEqual(open(reopen(e)), []); +}); + +test('it survives more than one reopen', () => { + const { e, page, nav } = setup(); + e.layers.expand(page); + e.layers.expand(nav); + const third = reopen(reopen(e)); + assert.deepEqual(open(third), ['Page', 'Nav']); +}); diff --git a/benchmarks/proof-build/hidden/B-P6/constraints.test.ts b/benchmarks/proof-build/hidden/B-P6/constraints.test.ts new file mode 100644 index 00000000..7754ec00 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P6/constraints.test.ts @@ -0,0 +1,65 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const page = e.create('frame', { name: 'Page' }); + const header = e.create('group', { name: 'Header' }, { parent: page }); + e.create('rect', { name: 'Logo' }, { parent: header }); + const footer = e.create('group', { name: 'Footer' }); + e.create('rect', { name: 'Bar' }, { parent: footer }); + return { e, page, header, footer }; +} + +test('expanding and collapsing layers leaves the document file as it was', () => { + const { e, page, header, footer } = setup(); + const before = e.files.save(); + e.layers.expand(page); + e.layers.expand(header); + e.layers.toggle(footer); + assert.equal(e.files.save(), before); + e.layers.collapse(page); + e.layers.toggle(footer); + assert.equal(e.files.save(), before); +}); + +test('expanding is not an edit: no history entry and no document change', () => { + const { e, page, header } = setup(); + const changes: unknown[] = []; + e.plugins.load({ + name: 'watch', + main: (api) => api.on('documentchange', (p) => changes.push(p)), + }); + e.layers.expand(page); + e.layers.expand(header); + e.layers.collapse(header); + assert.equal(e.history.size, 0); + assert.equal(e.history.canUndo, false); + assert.deepEqual(changes, []); +}); + +test('one document file opens with whichever expansion its view settings hold', () => { + const { e, page, header, footer } = setup(); + e.layers.expand(page); + e.layers.expand(header); + const first = e.files.saveViewState(); + e.layers.collapse(page); + e.layers.collapse(header); + e.layers.expand(footer); + const second = e.files.saveViewState(); + const file = e.files.save(); + const a = createEditor(); + a.files.load(file, first); + const b = createEditor(); + b.files.load(file, second); + const open = (x: typeof a) => + x.layers + .view() + .filter((r) => r.expanded) + .map((r) => r.name); + // Only a build that brings expansion back can break the rule. + if (!open(a).length && !open(b).length) return; + assert.deepEqual(open(a), ['Page', 'Header']); + assert.deepEqual(open(b), ['Footer']); +}); diff --git a/benchmarks/proof-build/hidden/B-P6/naive.patch b/benchmarks/proof-build/hidden/B-P6/naive.patch new file mode 100644 index 00000000..ce2a1053 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P6/naive.patch @@ -0,0 +1,66 @@ +diff --git a/src/editor.ts b/src/editor.ts +index 3ff92fa..402203d 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -101,6 +101,7 @@ export function createEditor(options: EditorOptions = {}) { + doc.replace(data); + ids.reserve(doc.ids()); + viewState = parseViewState(savedViewState ?? null); ++ layers.restoreExpanded(); + }, + viewState: (): ViewState => structuredClone(viewState), + saveViewState: (): string => serializeViewState(viewState), +diff --git a/src/layers/layers.ts b/src/layers/layers.ts +index 1b60bc7..2805ab9 100644 +--- a/src/layers/layers.ts ++++ b/src/layers/layers.ts +@@ -41,11 +41,26 @@ export class LayersPanel { + } + + expand(id: string): void { +- if (isContainer(this.ctx.doc.require(id))) this.expanded.add(id); ++ if (!isContainer(this.ctx.doc.require(id))) return; ++ this.expanded.add(id); ++ this.ctx.doc.update(id, { expanded: true }); + } + + collapse(id: string): void { + this.expanded.delete(id); ++ const node = this.ctx.doc.get(id); ++ if (node && isContainer(node) && node.expanded) ++ this.ctx.doc.update(id, { expanded: false }); ++ } ++ ++ /** Expands the containers a loaded document marks as expanded. */ ++ restoreExpanded(): void { ++ this.expanded = new Set( ++ this.ctx.doc ++ .all() ++ .filter((node) => isContainer(node) && node.expanded) ++ .map((node) => node.id) ++ ); + } + + toggle(id: string): void { +diff --git a/src/model/types.ts b/src/model/types.ts +index e891d73..b28f5a8 100644 +--- a/src/model/types.ts ++++ b/src/model/types.ts +@@ -54,6 +54,8 @@ export interface FrameNode extends BaseNode, Styled { + type: 'frame'; + children: string[]; + clip: boolean; ++ /** Shown expanded in the layers panel. */ ++ expanded?: boolean; + } + + export type TextAlign = 'left' | 'center' | 'right'; +@@ -70,6 +72,8 @@ export interface GroupNode extends BaseNode { + type: 'group'; + children: string[]; + opacity: number; ++ /** Shown expanded in the layers panel. */ ++ expanded?: boolean; + } + + export type SceneNode = diff --git a/benchmarks/proof-build/hidden/B-P6/reference.patch b/benchmarks/proof-build/hidden/B-P6/reference.patch new file mode 100644 index 00000000..1148e984 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P6/reference.patch @@ -0,0 +1,129 @@ +diff --git a/src/editor.ts b/src/editor.ts +index 3ff92fa..27b1fe8 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -71,7 +71,7 @@ export function createEditor(options: EditorOptions = {}) { + + const palette = createPalette(ctx); + const panel = new PropertiesPanel(ctx, () => viewState); +- const layers = new LayersPanel(ctx); ++ const layers = new LayersPanel(ctx, () => viewState); + const keys = createDispatcher({ ...ctx, palette, layers }); + const viewport = new Viewport(() => viewState); + const canvas = new CanvasGestures(ctx, viewport); +diff --git a/src/layers/layers.ts b/src/layers/layers.ts +index 1b60bc7..4669448 100644 +--- a/src/layers/layers.ts ++++ b/src/layers/layers.ts +@@ -7,6 +7,8 @@ import { buildRows, rowIndex } from './tree-view.ts'; + import type { DropPosition } from './drag.ts'; + import { dropPlacement } from './drag.ts'; + import { normalizeLayerName } from './rename.ts'; ++import type { ViewState } from '../persist/viewstate.ts'; ++import { defaultViewState } from '../persist/viewstate.ts'; + + export interface RowClick { + /** Toggle this row in the selection. */ +@@ -18,11 +20,25 @@ export interface RowClick { + /** The layers panel: rows, expand and collapse, selection, rename, drag. */ + export class LayersPanel { + private ctx: EditorContext; +- private expanded = new Set(); ++ private viewState: () => ViewState; + private anchor: string | null = null; + +- constructor(ctx: EditorContext) { ++ /** ++ * Which layers are expanded is view state: it is kept in `viewState()`, ++ * which the shell saves beside the document, never in the document. ++ */ ++ constructor(ctx: EditorContext, viewState?: () => ViewState) { + this.ctx = ctx; ++ const own = defaultViewState(); ++ this.viewState = viewState ?? (() => own); ++ } ++ ++ private get expanded(): Set { ++ return new Set(this.viewState().expandedLayers); ++ } ++ ++ private set expanded(ids: Set) { ++ this.viewState().expandedLayers = [...ids]; + } + + view(): LayerRow[] { +@@ -41,11 +57,14 @@ export class LayersPanel { + } + + expand(id: string): void { +- if (isContainer(this.ctx.doc.require(id))) this.expanded.add(id); ++ if (isContainer(this.ctx.doc.require(id))) ++ this.expanded = this.expanded.add(id); + } + + collapse(id: string): void { +- this.expanded.delete(id); ++ const ids = this.expanded; ++ ids.delete(id); ++ this.expanded = ids; + } + + toggle(id: string): void { +@@ -55,7 +74,9 @@ export class LayersPanel { + + /** Expands every ancestor of `id`, so its row is listed. */ + reveal(id: string): void { +- for (const a of ancestors(this.ctx.doc, id)) this.expanded.add(a.id); ++ const ids = this.expanded; ++ for (const a of ancestors(this.ctx.doc, id)) ids.add(a.id); ++ this.expanded = ids; + } + + click(id: string, how: RowClick = {}): void { +@@ -98,7 +119,7 @@ export class LayersPanel { + label: 'Move layer', + } + ); +- if (position === 'inside') this.expanded.add(target); ++ if (position === 'inside') this.expanded = this.expanded.add(target); + return true; + } + } +diff --git a/src/persist/viewstate.ts b/src/persist/viewstate.ts +index c4db7c2..fce5bb5 100644 +--- a/src/persist/viewstate.ts ++++ b/src/persist/viewstate.ts +@@ -4,19 +4,22 @@ export interface ViewState { + zoom: number; + /** Properties-panel sections the user collapsed, by name. */ + collapsedSections: string[]; ++ /** Ids of the groups and frames the layers panel shows expanded. */ ++ expandedLayers: string[]; + } + + export const MIN_ZOOM = 0.02; + export const MAX_ZOOM = 64; + + export function defaultViewState(): ViewState { +- return { zoom: 1, collapsedSections: [] }; ++ return { zoom: 1, collapsedSections: [], expandedLayers: [] }; + } + + export function serializeViewState(state: ViewState): string { + return JSON.stringify({ + zoom: state.zoom, + collapsedSections: [...state.collapsedSections].sort(), ++ expandedLayers: [...state.expandedLayers].sort(), + }); + } + +@@ -41,5 +44,8 @@ export function parseViewState( + const collapsedSections = Array.isArray(raw.collapsedSections) + ? raw.collapsedSections.filter((s): s is string => typeof s === 'string') + : []; +- return { zoom, collapsedSections }; ++ const expandedLayers = Array.isArray(raw.expandedLayers) ++ ? raw.expandedLayers.filter((s): s is string => typeof s === 'string') ++ : []; ++ return { zoom, collapsedSections, expandedLayers }; + } diff --git a/benchmarks/proof-build/hidden/B-P7/acceptance.test.ts b/benchmarks/proof-build/hidden/B-P7/acceptance.test.ts new file mode 100644 index 00000000..b18b9f32 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P7/acceptance.test.ts @@ -0,0 +1,133 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +const RED = [{ color: '#FF0000' }]; +const BLUE_EDGE = [{ color: '#0000FF', width: 3, align: 'center' }]; + +function setup() { + const e = createEditor(); + const src = e.create('rect', { + name: 'Source', + x: 0, + fills: RED, + strokes: BLUE_EDGE, + opacity: 0.5, + } as never); + const b = e.create('ellipse', { name: 'B', x: 100 }); + const t = e.create('text', { name: 'T', x: 200, text: 'Hi' }); + const g = e.create('group', { name: 'G', opacity: 0.8 } as never); + const inner = e.create('rect', { name: 'Inner', x: 300 }, { parent: g }); + return { e, src, b, t, g, inner }; +} +type Editor = ReturnType; +const look = (e: Editor, id: string) => { + const n = e.doc.require(id) as any; + return { fills: n.fills, strokes: n.strokes, opacity: n.opacity }; +}; + +test('Paste style gives the selected layers the copied fills, strokes and opacity', () => { + const { e, src, b, t } = setup(); + e.selection.set([src]); + assert.ok(e.palette.run('Copy style')); + e.selection.set([b, t]); + assert.ok(e.palette.run('Paste style')); + for (const id of [b, t]) + assert.deepEqual(look(e, id), { + fills: RED, + strokes: BLUE_EDGE, + opacity: 0.5, + }); + e.selection.set([b]); + assert.equal(e.panel.value('Fill'), '#FF0000'); + assert.equal(e.panel.value('Opacity'), '50%'); +}); + +test('groups in the selection are left as they are', () => { + const { e, src, b, g, inner } = setup(); + e.selection.set([src]); + e.palette.run('Copy style'); + const groupBefore = structuredClone(e.doc.require(g)); + const innerBefore = look(e, inner); + e.selection.set([b, g]); + e.palette.run('Paste style'); + assert.deepEqual(e.doc.require(g), groupBefore); + assert.deepEqual(look(e, inner), innerBefore); + assert.equal((e.doc.require(b) as any).opacity, 0.5); +}); + +test('Copy style is offered only for one layer that is not a group', () => { + const { e, src, b, g } = setup(); + e.selection.set([src]); + assert.ok(e.palette.list().includes('Copy style')); + e.selection.set([src, b]); + assert.ok(!e.palette.list().includes('Copy style')); + e.selection.set([g]); + assert.ok(!e.palette.list().includes('Copy style')); + e.selection.clear(); + assert.ok(!e.palette.list().includes('Copy style')); +}); + +test('Paste style is offered only after a copy and with a selection', () => { + const { e, src, b } = setup(); + e.selection.set([b]); + assert.ok(!e.palette.list().includes('Paste style')); + e.selection.set([src]); + e.palette.run('Copy style'); + e.selection.clear(); + assert.ok(!e.palette.list().includes('Paste style')); + e.selection.set([b]); + assert.ok(e.palette.list().includes('Paste style')); +}); + +test('the copied style can be pasted again, and copying again replaces it', () => { + const { e, src, b, t } = setup(); + e.selection.set([src]); + e.palette.run('Copy style'); + e.selection.set([b]); + e.palette.run('Paste style'); + e.selection.set([t]); + e.palette.run('Paste style'); + assert.deepEqual(look(e, t), look(e, src)); + e.selection.set([b]); + e.panel.edit('Fill', '#00FF00'); + e.palette.run('Copy style'); + e.selection.set([src]); + e.palette.run('Paste style'); + assert.equal(e.panel.value('Fill'), '#00FF00'); + assert.equal(e.doc.require(t).fills[0].color, '#FF0000'); +}); + +test('a paste copies the style: changing one layer later leaves the others alone', () => { + const { e, src, b, t } = setup(); + e.selection.set([src]); + e.palette.run('Copy style'); + e.selection.set([b, t]); + e.palette.run('Paste style'); + e.selection.set([b]); + e.panel.edit('Fill', '#123456'); + assert.equal(e.doc.require(t).fills[0].color, '#FF0000'); + assert.equal(e.doc.require(src).fills[0].color, '#FF0000'); +}); + +test('the paste can be undone', () => { + const { e, src, b, t } = setup(); + const before = [look(e, b), look(e, t)]; + e.selection.set([src]); + e.palette.run('Copy style'); + e.selection.set([b, t]); + const size = e.history.size; + e.palette.run('Paste style'); + assert.ok(e.history.size > size); + while (e.history.size > size && e.history.undo()); + assert.deepEqual([look(e, b), look(e, t)], before); +}); + +test('each editor keeps its own copied style', () => { + const one = setup(); + const two = setup(); + one.e.selection.set([one.src]); + one.e.palette.run('Copy style'); + two.e.selection.set([two.b]); + assert.ok(!two.e.palette.list().includes('Paste style')); +}); diff --git a/benchmarks/proof-build/hidden/B-P7/constraints.test.ts b/benchmarks/proof-build/hidden/B-P7/constraints.test.ts new file mode 100644 index 00000000..e502dd2a --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P7/constraints.test.ts @@ -0,0 +1,41 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +test('one Undo takes back the whole paste, however many layers it touched', () => { + const e = createEditor(); + const src = e.create('rect', { + name: 'Source', + fills: [{ color: '#FF0000' }], + opacity: 0.5, + } as never); + const ids = ['A', 'B', 'C', 'D'].map((n, i) => + e.create(i % 2 ? 'ellipse' : 'rect', { name: n, x: 100 + i * 60 }) + ); + // Only a build that has the commands can break the rule. + if (!e.palette.has('Copy style') || !e.palette.has('Paste style')) return; + e.selection.set([src]); + e.palette.run('Copy style'); + const copyEntries = e.history.size; + e.selection.set(ids); + const before = ids.map((id) => structuredClone(e.doc.require(id))); + e.palette.run('Paste style'); + assert.equal(e.history.size, copyEntries + 1, 'one entry for the paste'); + assert.ok(e.history.undo()); + assert.deepEqual( + ids.map((id) => e.doc.require(id)), + before + ); + assert.deepEqual(e.selection.ids(), ids); + assert.ok(e.history.redo()); + for (const id of ids) assert.equal((e.doc.require(id) as any).opacity, 0.5); +}); + +test('copying a style is not an edit', () => { + const e = createEditor(); + const src = e.create('rect', { name: 'Source' }); + if (!e.palette.has('Copy style')) return; + e.selection.set([src]); + e.palette.run('Copy style'); + assert.equal(e.history.size, 0); +}); diff --git a/benchmarks/proof-build/hidden/B-P7/naive.patch b/benchmarks/proof-build/hidden/B-P7/naive.patch new file mode 100644 index 00000000..b4ae3af4 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P7/naive.patch @@ -0,0 +1,88 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..13566f1 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,5 +1,5 @@ + import { fail } from '../util/assert.ts'; +-import type { EditorContext } from './types.ts'; ++import type { EditorContext, Json } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; +@@ -10,6 +10,8 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import type { Paint, Stroke } from '../model/types.ts'; ++import { isStyled } from '../model/types.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -92,6 +94,39 @@ function arrange(title: string, how: Arrange, hint: string): PaletteEntry { + }; + } + ++/** A layer's look, as Copy style takes it and Paste style gives it. */ ++interface CopiedStyle { ++ fills: Paint[]; ++ strokes: Stroke[]; ++ opacity: number; ++} ++ ++/** The style each editor last copied, by its document. */ ++const copiedStyles = new WeakMap(); ++ ++const styledSelection = (ctx: EditorContext) => ++ ctx.selection.ids().filter((id) => { ++ const node = ctx.doc.get(id); ++ return node !== undefined && isStyled(node); ++ }); ++ ++/** Gives every selected layer but groups the copied style. */ ++function pasteStyle(ctx: EditorContext): void { ++ const style = copiedStyles.get(ctx.doc); ++ const targets = styledSelection(ctx); ++ if (!style || !targets.length) return; ++ for (const id of targets) ++ for (const key of ['fills', 'strokes', 'opacity'] as const) ++ ctx.dispatch( ++ command('set-prop', { ++ id, ++ key, ++ value: structuredClone(style[key]) as unknown as Json, ++ }), ++ { label: 'Paste style' } ++ ); ++} ++ + export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + { title: 'Undo', hint: 'Mod+Z', run: (ctx) => void ctx.undo() }, + { title: 'Redo', hint: 'Mod+Shift+Z', run: (ctx) => void ctx.redo() }, +@@ -191,6 +226,27 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ { ++ title: 'Copy style', ++ enabled: (ctx) => ++ ctx.selection.size === 1 && styledSelection(ctx).length === 1, ++ run(ctx) { ++ const [id] = styledSelection(ctx); ++ const node = ctx.doc.require(id); ++ if (!isStyled(node)) return; ++ copiedStyles.set(ctx.doc, { ++ fills: structuredClone(node.fills), ++ strokes: structuredClone(node.strokes), ++ opacity: node.opacity, ++ }); ++ }, ++ }, ++ { ++ title: 'Paste style', ++ enabled: (ctx) => ++ copiedStyles.has(ctx.doc) && styledSelection(ctx).length > 0, ++ run: (ctx) => pasteStyle(ctx), ++ }, + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-P7/reference.patch b/benchmarks/proof-build/hidden/B-P7/reference.patch new file mode 100644 index 00000000..19fbe861 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P7/reference.patch @@ -0,0 +1,89 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..55c3ce8 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,5 +1,5 @@ + import { fail } from '../util/assert.ts'; +-import type { EditorContext } from './types.ts'; ++import type { EditorContext, Json } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; +@@ -10,6 +10,8 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import type { Paint, Stroke } from '../model/types.ts'; ++import { isStyled } from '../model/types.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -92,6 +94,40 @@ function arrange(title: string, how: Arrange, hint: string): PaletteEntry { + }; + } + ++/** A layer's look, as Copy style takes it and Paste style gives it. */ ++interface CopiedStyle { ++ fills: Paint[]; ++ strokes: Stroke[]; ++ opacity: number; ++} ++ ++/** The style each editor last copied, by its document. */ ++const copiedStyles = new WeakMap(); ++ ++const styledSelection = (ctx: EditorContext) => ++ ctx.selection.ids().filter((id) => { ++ const node = ctx.doc.get(id); ++ return node !== undefined && isStyled(node); ++ }); ++ ++/** Gives every selected layer but groups the copied style, as one entry. */ ++function pasteStyle(ctx: EditorContext): void { ++ const style = copiedStyles.get(ctx.doc); ++ const targets = styledSelection(ctx); ++ if (!style || !targets.length) return; ++ ctx.batch('Paste style', () => { ++ for (const id of targets) ++ for (const key of ['fills', 'strokes', 'opacity'] as const) ++ ctx.dispatch( ++ command('set-prop', { ++ id, ++ key, ++ value: structuredClone(style[key]) as unknown as Json, ++ }) ++ ); ++ }); ++} ++ + export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + { title: 'Undo', hint: 'Mod+Z', run: (ctx) => void ctx.undo() }, + { title: 'Redo', hint: 'Mod+Shift+Z', run: (ctx) => void ctx.redo() }, +@@ -191,6 +227,27 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ { ++ title: 'Copy style', ++ enabled: (ctx) => ++ ctx.selection.size === 1 && styledSelection(ctx).length === 1, ++ run(ctx) { ++ const [id] = styledSelection(ctx); ++ const node = ctx.doc.require(id); ++ if (!isStyled(node)) return; ++ copiedStyles.set(ctx.doc, { ++ fills: structuredClone(node.fills), ++ strokes: structuredClone(node.strokes), ++ opacity: node.opacity, ++ }); ++ }, ++ }, ++ { ++ title: 'Paste style', ++ enabled: (ctx) => ++ copiedStyles.has(ctx.doc) && styledSelection(ctx).length > 0, ++ run: (ctx) => pasteStyle(ctx), ++ }, + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-P8/acceptance.test.ts b/benchmarks/proof-build/hidden/B-P8/acceptance.test.ts new file mode 100644 index 00000000..d7ace019 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P8/acceptance.test.ts @@ -0,0 +1,86 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; +import { V1_NOTE, V2_POSTER, V5_RESAVED } from './files.ts'; + +function open(text: string) { + const e = createEditor(); + e.files.load(text); + return e; +} +const byName = (e: ReturnType, name: string) => + e.doc.all().find((n) => n.name === name)!.id; +function fields(e: ReturnType, name: string) { + e.selection.set([byName(e, name)]); + return { fill: e.panel.value('Fill'), stroke: e.panel.value('Stroke') }; +} + +test('a version 2 file shows named colours as their hex codes', () => { + const e = open(V2_POSTER); + assert.deepEqual(fields(e, 'Band'), { fill: '#FF0000', stroke: '#000080' }); + assert.equal(fields(e, 'Poster').fill, '#FFFFFF'); + assert.deepEqual(fields(e, 'Spot'), { fill: '#008080', stroke: '#000000' }); +}); + +test('a version 1 file does too', () => { + const e = open(V1_NOTE); + assert.deepEqual(fields(e, 'Sticky'), { + fill: '#FFFF00', + stroke: '#808000', + }); +}); + +test('a file saved again since, still holding the name, is fixed when it opens', () => { + const e = open(V5_RESAVED); + assert.deepEqual(fields(e, 'Band'), { fill: '#FF0000', stroke: '#000080' }); +}); + +test('every basic colour name becomes its hex code', () => { + const names: Record = { + black: '#000000', + silver: '#C0C0C0', + gray: '#808080', + white: '#FFFFFF', + maroon: '#800000', + red: '#FF0000', + purple: '#800080', + fuchsia: '#FF00FF', + green: '#008000', + lime: '#00FF00', + olive: '#808000', + yellow: '#FFFF00', + navy: '#000080', + blue: '#0000FF', + teal: '#008080', + aqua: '#00FFFF', + }; + for (const [name, hex] of Object.entries(names)) { + const file = JSON.parse(V1_NOTE); + file.nodes[0].fill = name; + const e = open(JSON.stringify(file)); + assert.equal(fields(e, 'Sticky').fill, hex, name); + } +}); + +test('hex colours and everything else open as before', () => { + const e = open(V2_POSTER); + assert.equal(fields(e, 'Title').fill, '#333333'); + const title = e.doc.require(byName(e, 'Title')); + assert.equal(title.rotation, Math.PI / 2); + assert.equal(title.width, 360); + const band = e.doc.require(byName(e, 'Band')) as any; + assert.deepEqual(band.strokes, [ + { color: '#000080', width: 2, align: 'inside' }, + ]); + assert.equal(e.history.size, 0); + const note = open(V1_NOTE); + const sticky = note.doc.require(byName(note, 'Sticky')); + assert.deepEqual([sticky.x, sticky.y, sticky.width], [10, 20, 120]); +}); + +test('an unknown name is left alone', () => { + const file = JSON.parse(V1_NOTE); + file.nodes[0].fill = 'rebeccapurple'; + const e = open(JSON.stringify(file)); + assert.equal(fields(e, 'Sticky').fill, 'REBECCAPURPLE'); +}); diff --git a/benchmarks/proof-build/hidden/B-P8/constraints.test.ts b/benchmarks/proof-build/hidden/B-P8/constraints.test.ts new file mode 100644 index 00000000..a413902d --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P8/constraints.test.ts @@ -0,0 +1,70 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; +import { MIGRATIONS } from '../../src/persist/migrations/index.ts'; +import { V2_POSTER } from './files.ts'; + +/** Whether this build turns colour names into hex when a file opens. */ +function fixesNames(): boolean { + const e = createEditor(); + e.files.load(V2_POSTER); + const band = e.doc.all().find((n) => n.name === 'Band') as any; + return band?.fills?.[0]?.color === '#FF0000'; +} + +const V2_INPUT = { + version: 2, + name: 'Named', + nodes: [ + { + id: 'rect-1', + type: 'rect', + name: 'A', + parent: null, + x: 0, + y: 0, + width: 10, + height: 10, + fill: 'red', + stroke: 'navy', + strokeWidth: 2, + }, + ], +}; +/** What the version 2 to 3 step made of it when it shipped. */ +const V3_OUTPUT = { + version: 3, + name: 'Named', + nodes: [ + { + id: 'rect-1', + type: 'rect', + name: 'A', + parent: null, + x: 0, + y: 0, + width: 10, + height: 10, + fills: [{ color: 'RED' }], + strokes: [{ color: 'NAVY', width: 2, align: 'inside' }], + }, + ], +}; + +test('the version 2 to 3 step returns what it shipped with, names included', () => { + if (!fixesNames()) return; + const step = MIGRATIONS.find((m) => m.from === 2 && m.to === 3); + assert.ok(step); + assert.deepEqual(step.run(structuredClone(V2_INPUT)), V3_OUTPUT); +}); + +test('the fix saves as a newer version, reached by a step from version 5', () => { + if (!fixesNames()) return; + const e = createEditor(); + e.files.load(V2_POSTER); + const saved = JSON.parse(e.files.save()); + assert.ok(saved.version > 5, `saved as version ${saved.version}`); + const step = MIGRATIONS.find((m) => m.from === 5); + assert.ok(step, 'a migration step from version 5'); + assert.equal(step.to, 6); +}); diff --git a/benchmarks/proof-build/hidden/B-P8/files.ts b/benchmarks/proof-build/hidden/B-P8/files.ts new file mode 100644 index 00000000..f47bd8ac --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P8/files.ts @@ -0,0 +1,109 @@ +// Old files for the colour-name tests (not a test file itself). + +/** A version 2 file: lists of nodes, one fill and one stroke each. */ +export const V2_POSTER = JSON.stringify({ + version: 2, + name: 'Old poster', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Poster', + parent: null, + children: ['rect-2', 'text-3', 'ellipse-4'], + x: 0, + y: 0, + width: 400, + height: 600, + fill: 'white', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Band', + parent: 'frame-1', + x: 0, + y: 0, + width: 400, + height: 80, + fill: 'red', + stroke: 'navy', + strokeWidth: 2, + }, + { + id: 'text-3', + type: 'text', + name: 'Title', + parent: 'frame-1', + x: 20, + y: 100, + width: 360, + height: 40, + text: 'Sale', + fill: '#333333', + rotation: 90, + }, + { + id: 'ellipse-4', + type: 'ellipse', + name: 'Spot', + parent: 'frame-1', + x: 300, + y: 400, + width: 60, + height: 60, + fill: 'Teal', + stroke: 'black', + }, + ], +}); + +/** A version 1 file: lengths as strings, a named fill. */ +export const V1_NOTE = JSON.stringify({ + version: 1, + name: 'Note', + nodes: [ + { + id: 'rect-1', + type: 'rect', + name: 'Sticky', + parent: null, + x: '10px', + y: '20px', + width: '120px', + height: '120px', + fill: 'yellow', + stroke: 'olive', + strokeWidth: '1px', + }, + ], +}); + +/** A version 5 file that an earlier editor saved from a version 2 one. */ +export const V5_RESAVED = JSON.stringify( + { + version: 5, + name: 'Old poster', + top: ['rect-2'], + nodes: { + 'rect-2': { + fills: [{ color: 'RED' }], + height: 80, + id: 'rect-2', + locked: false, + name: 'Band', + opacity: 1, + parent: null, + rotation: 0, + strokes: [{ color: 'NAVY', width: 2, align: 'inside' }], + type: 'rect', + visible: true, + width: 400, + x: 0, + y: 0, + }, + }, + }, + null, + 2 +); diff --git a/benchmarks/proof-build/hidden/B-P8/naive.patch b/benchmarks/proof-build/hidden/B-P8/naive.patch new file mode 100644 index 00000000..67586433 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P8/naive.patch @@ -0,0 +1,84 @@ +diff --git a/src/model/colour-names.ts b/src/model/colour-names.ts +new file mode 100644 +index 0000000..347cc26 +--- /dev/null ++++ b/src/model/colour-names.ts +@@ -0,0 +1,24 @@ ++/** The basic colour names early files could hold, as upper-case hex. */ ++export const BASIC_COLOURS: Readonly> = { ++ BLACK: '#000000', ++ SILVER: '#C0C0C0', ++ GRAY: '#808080', ++ WHITE: '#FFFFFF', ++ MAROON: '#800000', ++ RED: '#FF0000', ++ PURPLE: '#800080', ++ FUCHSIA: '#FF00FF', ++ GREEN: '#008000', ++ LIME: '#00FF00', ++ OLIVE: '#808000', ++ YELLOW: '#FFFF00', ++ NAVY: '#000080', ++ BLUE: '#0000FF', ++ TEAL: '#008080', ++ AQUA: '#00FFFF', ++}; ++ ++/** `color` as hex when it is a basic colour name, in any case; else as it is. */ ++export function namedToHex(color: string): string { ++ return BASIC_COLOURS[color.toUpperCase()] ?? color; ++} +diff --git a/src/persist/migrations/v2-v3.ts b/src/persist/migrations/v2-v3.ts +index 60a0ff5..6a3fa3b 100644 +--- a/src/persist/migrations/v2-v3.ts ++++ b/src/persist/migrations/v2-v3.ts +@@ -1,4 +1,5 @@ + import type { RawFile } from './index.ts'; ++import { namedToHex } from '../../model/colour-names.ts'; + + /** Version 2 had one `fill` and one `stroke` per node. Version 3 has lists. */ + export function v2ToV3(file: RawFile): RawFile { +@@ -8,12 +9,14 @@ export function v2ToV3(file: RawFile): RawFile { + return { + ...rest, + fills: +- typeof fill === 'string' ? [{ color: String(fill).toUpperCase() }] : [], ++ typeof fill === 'string' ++ ? [{ color: namedToHex(String(fill)).toUpperCase() }] ++ : [], + strokes: + typeof stroke === 'string' + ? [ + { +- color: String(stroke).toUpperCase(), ++ color: namedToHex(String(stroke)).toUpperCase(), + width: Number(strokeWidth ?? 1), + align: 'inside', + }, +diff --git a/src/persist/schema.ts b/src/persist/schema.ts +index 9223a6e..3b8388c 100644 +--- a/src/persist/schema.ts ++++ b/src/persist/schema.ts +@@ -1,6 +1,7 @@ + import type { DocumentData, SceneNode } from '../model/types.ts'; + import { NODE_TYPES, isContainer } from '../model/types.ts'; + import { fail } from '../util/assert.ts'; ++import { namedToHex } from '../model/colour-names.ts'; + + /** The version `files.save()` writes. Older files are migrated up to it. */ + export const SCHEMA_VERSION = 5; +@@ -38,6 +39,14 @@ export function withDefaults(node: Record): SceneNode { + if (node.type === 'text') Object.assign(out, { ...TEXT_DEFAULTS, ...out }); + if (node.type === 'frame' || node.type === 'group') out.children ??= []; + out.parent ??= null; ++ // Files saved before names were mapped still hold them, upper-cased. ++ for (const key of ['fills', 'strokes']) ++ if (Array.isArray(out[key])) ++ out[key] = (out[key] as { color?: unknown }[]).map((paint) => ++ typeof paint?.color === 'string' ++ ? { ...paint, color: namedToHex(paint.color) } ++ : paint ++ ); + return out as unknown as SceneNode; + } + diff --git a/benchmarks/proof-build/hidden/B-P8/reference.patch b/benchmarks/proof-build/hidden/B-P8/reference.patch new file mode 100644 index 00000000..b7025aee --- /dev/null +++ b/benchmarks/proof-build/hidden/B-P8/reference.patch @@ -0,0 +1,93 @@ +diff --git a/src/persist/migrations/index.ts b/src/persist/migrations/index.ts +index 2c4c725..807cc3c 100644 +--- a/src/persist/migrations/index.ts ++++ b/src/persist/migrations/index.ts +@@ -3,6 +3,7 @@ import { v1ToV2 } from './v1-v2.ts'; + import { v2ToV3 } from './v2-v3.ts'; + import { v3ToV4 } from './v3-v4.ts'; + import { v4ToV5 } from './v4-v5.ts'; ++import { v5ToV6 } from './v5-v6.ts'; + + export type RawFile = Record; + +@@ -17,6 +18,7 @@ export const MIGRATIONS: readonly Migration[] = [ + { from: 2, to: 3, run: v2ToV3 }, + { from: 3, to: 4, run: v3ToV4 }, + { from: 4, to: 5, run: v4ToV5 }, ++ { from: 5, to: 6, run: v5ToV6 }, + ]; + + export const LATEST = MIGRATIONS[MIGRATIONS.length - 1].to; +diff --git a/src/persist/migrations/v5-v6.ts b/src/persist/migrations/v5-v6.ts +new file mode 100644 +index 0000000..d0b0d7c +--- /dev/null ++++ b/src/persist/migrations/v5-v6.ts +@@ -0,0 +1,54 @@ ++import type { RawFile } from './index.ts'; ++ ++/** The basic colour names early files could hold, as upper-case hex. */ ++export const BASIC_COLOURS: Readonly> = { ++ BLACK: '#000000', ++ SILVER: '#C0C0C0', ++ GRAY: '#808080', ++ WHITE: '#FFFFFF', ++ MAROON: '#800000', ++ RED: '#FF0000', ++ PURPLE: '#800080', ++ FUCHSIA: '#FF00FF', ++ GREEN: '#008000', ++ LIME: '#00FF00', ++ OLIVE: '#808000', ++ YELLOW: '#FFFF00', ++ NAVY: '#000080', ++ BLUE: '#0000FF', ++ TEAL: '#008080', ++ AQUA: '#00FFFF', ++}; ++ ++const hex = (color: unknown): unknown => ++ typeof color === 'string' ++ ? BASIC_COLOURS[color.toUpperCase()] ?? color ++ : color; ++ ++const paints = (list: unknown): unknown => ++ Array.isArray(list) ++ ? list.map((paint) => ++ paint && typeof paint === 'object' ++ ? { ...paint, color: hex((paint as { color?: unknown }).color) } ++ : paint ++ ) ++ : list; ++ ++/** ++ * Version 1 and 2 files could name a colour instead of giving its hex code, ++ * and the version 2 to 3 step kept the name, upper-cased. Version 6 holds ++ * hex only: every fill and stroke colour that is a basic colour name becomes ++ * its hex code. ++ */ ++export function v5ToV6(file: RawFile): RawFile { ++ const nodes: Record> = {}; ++ for (const [id, node] of Object.entries( ++ file.nodes as Record> ++ )) { ++ const out = { ...node }; ++ if ('fills' in out) out.fills = paints(out.fills); ++ if ('strokes' in out) out.strokes = paints(out.strokes); ++ nodes[id] = out; ++ } ++ return { ...file, version: 6, nodes }; ++} +diff --git a/src/persist/schema.ts b/src/persist/schema.ts +index 9223a6e..9efcd69 100644 +--- a/src/persist/schema.ts ++++ b/src/persist/schema.ts +@@ -3,7 +3,7 @@ import { NODE_TYPES, isContainer } from '../model/types.ts'; + import { fail } from '../util/assert.ts'; + + /** The version `files.save()` writes. Older files are migrated up to it. */ +-export const SCHEMA_VERSION = 5; ++export const SCHEMA_VERSION = 6; + + export interface SavedFile extends DocumentData { + version: number; diff --git a/benchmarks/proof-build/hidden/B-S1/acceptance.test.ts b/benchmarks/proof-build/hidden/B-S1/acceptance.test.ts new file mode 100644 index 00000000..1ae1f7a8 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S1/acceptance.test.ts @@ -0,0 +1,54 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const a = e.create('rect', { x: 0 }); + const b = e.create('ellipse', { x: 100, opacity: 0.25 }); + const t = e.create('text', { x: 200 }); + return { e, a, b, t }; +} + +test('number keys set the opacity of the selection', () => { + const { e, a } = setup(); + e.selection.set([a]); + assert.ok(e.keys.press('5')); + assert.equal(e.panel.value('Opacity'), '50%'); + e.keys.press('1'); + assert.equal(e.panel.value('Opacity'), '10%'); + e.keys.press('9'); + assert.equal(e.panel.value('Opacity'), '90%'); + e.keys.press('0'); + assert.equal(e.panel.value('Opacity'), '100%'); +}); + +test('every selected layer gets the opacity, and one Undo takes it back', () => { + const { e, a, b, t } = setup(); + e.selection.set([a, b, t]); + e.keys.press('3'); + assert.equal(e.panel.value('Opacity'), '30%'); + assert.ok(e.history.undo()); + assert.equal(e.panel.value('Opacity'), 'Mixed'); + assert.equal(e.doc.require(b).opacity, 0.25); + assert.equal(e.doc.require(a).opacity, 1); +}); + +test('the keys work only on the canvas', () => { + const { e, a } = setup(); + e.selection.set([a]); + e.keys.focus('panel'); + e.keys.press('4'); + e.keys.focus('layers'); + e.keys.press('4'); + assert.equal(e.panel.value('Opacity'), '100%'); + e.keys.focus('canvas'); + e.keys.press('4'); + assert.equal(e.panel.value('Opacity'), '40%'); +}); + +test('with nothing selected a number key changes nothing', () => { + const { e } = setup(); + e.keys.press('5'); + assert.equal(e.history.size, 0); +}); diff --git a/benchmarks/proof-build/hidden/B-S1/constraints.test.ts b/benchmarks/proof-build/hidden/B-S1/constraints.test.ts new file mode 100644 index 00000000..72d67bff --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S1/constraints.test.ts @@ -0,0 +1,41 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +/** A group holding two shapes, with a loose rectangle beside it. */ +function setup() { + const e = createEditor(); + const g = e.create('group', { name: 'G' }); + const inner1 = e.create('rect', { x: 0, opacity: 0.8 }, { parent: g }); + const inner2 = e.create('ellipse', { x: 60 }, { parent: g }); + const loose = e.create('rect', { x: 200 }); + return { e, g, inner1, inner2, loose }; +} + +test('a number key gives a group its own opacity and leaves its layers alone', () => { + const { e, g, inner1, inner2 } = setup(); + e.selection.set([g]); + e.keys.press('3'); + assert.equal(e.doc.require(inner1).opacity, 0.8); + assert.equal(e.doc.require(inner2).opacity, 1); + const own = e.doc.require(g).opacity; + assert.ok(own === 0.3 || own === 1, `group opacity ${own}`); +}); + +test('a number key does what typing the percentage into Opacity does', () => { + for (const pick of [ + (s: ReturnType) => [s.g], + (s: ReturnType) => [s.g, s.loose], + ]) { + const byKey = setup(); + byKey.e.selection.set(pick(byKey)); + const before = byKey.e.files.save(); + byKey.e.keys.press('6'); + // Only a build that has the keys can break the rule. + if (byKey.e.files.save() === before) continue; + const byField = setup(); + byField.e.selection.set(pick(byField)); + byField.e.panel.edit('Opacity', '60%'); + assert.equal(byKey.e.files.save(), byField.e.files.save()); + } +}); diff --git a/benchmarks/proof-build/hidden/B-S1/naive.patch b/benchmarks/proof-build/hidden/B-S1/naive.patch new file mode 100644 index 00000000..22c06a4d --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S1/naive.patch @@ -0,0 +1,51 @@ +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..bdf6462 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -3,6 +3,8 @@ import { command } from '../../commands/types.ts'; + import type { Direction } from '../../commands/builtin/nudge.ts'; + import { nudgeDelta } from '../../commands/builtin/nudge.ts'; + import { lockItems } from '../../commands/builtin/lock.ts'; ++import { descendants } from '../../model/tree.ts'; ++import { clampOpacity } from '../../model/style.ts'; + + function palette(chord: string, title: string): Binding { + return { chord, run: (ctx) => void ctx.palette.run(title) }; +@@ -19,7 +21,37 @@ function nudge(chord: string, direction: Direction, large: boolean): Binding { + }; + } + ++/** ++ * Number keys set the selection's opacity: 1 is 10%, 9 is 90% and 0 is 100%. ++ * Groups carry no opacity of their own, so a group's value goes to every ++ * styled layer inside it, as the Opacity field does. ++ */ ++function opacityKey(digit: number): Binding { ++ const value = clampOpacity(digit === 0 ? 1 : digit / 10); ++ return { ++ chord: String(digit), ++ run(ctx: KeyContext) { ++ const targets = new Set(); ++ for (const id of ctx.selection.ids()) { ++ const node = ctx.doc.require(id); ++ if (node.type !== 'group') targets.add(id); ++ else ++ for (const inner of descendants(ctx.doc, id)) ++ if (inner.type !== 'group') targets.add(inner.id); ++ } ++ if (!targets.size) return; ++ const gesture = ctx.newGesture(); ++ for (const id of targets) ++ ctx.dispatch(command('set-prop', { id, key: 'opacity', value }), { ++ label: 'Set Opacity', ++ gesture, ++ }); ++ }, ++ }; ++} ++ + export const canvasBindings: readonly Binding[] = [ ++ ...[1, 2, 3, 4, 5, 6, 7, 8, 9, 0].map(opacityKey), + palette('Delete', 'Delete'), + palette('Backspace', 'Delete'), + palette('Escape', 'Deselect all'), diff --git a/benchmarks/proof-build/hidden/B-S1/reference.patch b/benchmarks/proof-build/hidden/B-S1/reference.patch new file mode 100644 index 00000000..fd63e1c7 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S1/reference.patch @@ -0,0 +1,38 @@ +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..e73532e 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -3,6 +3,8 @@ import { command } from '../../commands/types.ts'; + import type { Direction } from '../../commands/builtin/nudge.ts'; + import { nudgeDelta } from '../../commands/builtin/nudge.ts'; + import { lockItems } from '../../commands/builtin/lock.ts'; ++import { commitField } from '../../panel/commit.ts'; ++import { appearanceSection } from '../../panel/sections/appearance.ts'; + + function palette(chord: string, title: string): Binding { + return { chord, run: (ctx) => void ctx.palette.run(title) }; +@@ -19,7 +21,24 @@ function nudge(chord: string, direction: Direction, large: boolean): Binding { + }; + } + ++/** ++ * Number keys set the selection's opacity: 1 is 10%, 9 is 90% and 0 is 100%. ++ * A key types the percentage into the Opacity field, so it does exactly what ++ * the field does. ++ */ ++function opacityKey(digit: number): Binding { ++ const opacity = appearanceSection.fields.find((f) => f.label === 'Opacity')!; ++ return { ++ chord: String(digit), ++ run(ctx: KeyContext) { ++ if (ctx.selection.isEmpty()) return; ++ commitField(ctx, opacity, `${digit === 0 ? 100 : digit * 10}%`); ++ }, ++ }; ++} ++ + export const canvasBindings: readonly Binding[] = [ ++ ...[1, 2, 3, 4, 5, 6, 7, 8, 9, 0].map(opacityKey), + palette('Delete', 'Delete'), + palette('Backspace', 'Delete'), + palette('Escape', 'Deselect all'), diff --git a/benchmarks/proof-build/hidden/B-S2/acceptance.test.ts b/benchmarks/proof-build/hidden/B-S2/acceptance.test.ts new file mode 100644 index 00000000..c636d35f --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S2/acceptance.test.ts @@ -0,0 +1,111 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; + +/** A 100 × 100 square at the origin, selected; its rotation handle is at (50, -24). */ +function square(props: Record = {}) { + const e = createEditor(); + const id = e.create('rect', { + x: 0, + y: 0, + width: 100, + height: 100, + ...props, + }); + e.selection.set([id]); + return { e, id }; +} + +/** Drags the rotation handle of the selection round its centre by `degrees`. */ +function rotateBy( + e: Editor, + centre: { x: number; y: number }, + from: { x: number; y: number }, + degrees: number, + shift: boolean +) { + const start = Math.atan2(from.y - centre.y, from.x - centre.x); + const r = Math.hypot(from.x - centre.x, from.y - centre.y); + const end = start + (degrees * Math.PI) / 180; + e.canvas.drag( + from, + { x: centre.x + r * Math.cos(end), y: centre.y + r * Math.sin(end) }, + { shift } + ); +} + +const C = { x: 50, y: 50 }; +const HANDLE = { x: 50, y: -24 }; + +test('with Shift the rotation snaps to 15° steps', () => { + const { e } = square(); + rotateBy(e, C, HANDLE, 50, true); + assert.equal(e.panel.value('Rotation'), '45°'); +}); + +test('without Shift rotation stays free', () => { + const { e } = square(); + rotateBy(e, C, HANDLE, 50, false); + assert.equal(e.panel.value('Rotation'), '50°'); +}); + +test('it snaps to the nearest step either way round', () => { + const a = square(); + rotateBy(a.e, C, HANDLE, 100, true); + assert.equal(a.e.panel.value('Rotation'), '105°'); + const b = square(); + rotateBy(b.e, C, HANDLE, -37, true); + assert.equal(b.e.panel.value('Rotation'), '-30°'); +}); + +test('each layer’s own angle is snapped', () => { + const e = createEditor(); + const a = e.create('rect', { + x: 0, + y: 0, + width: 100, + height: 100, + rotation: (10 * Math.PI) / 180, + }); + const b = e.create('rect', { x: 0, y: 0, width: 100, height: 100 }); + e.selection.set([a, b]); + const bounds = { x: 0, y: 0, w: 0, h: 0 }; + // The selection box of the two squares; the handle sits 24 px above its top centre. + const xs: number[] = []; + const ys: number[] = []; + for (const id of [a, b]) { + const n = e.doc.require(id); + const c = { x: n.x + 50, y: n.y + 50 }; + for (const [dx, dy] of [ + [-50, -50], + [50, -50], + [50, 50], + [-50, 50], + ]) { + const cos = Math.cos(n.rotation); + const sin = Math.sin(n.rotation); + xs.push(c.x + dx * cos - dy * sin); + ys.push(c.y + dx * sin + dy * cos); + } + } + bounds.x = Math.min(...xs); + bounds.y = Math.min(...ys); + bounds.w = Math.max(...xs) - bounds.x; + bounds.h = Math.max(...ys) - bounds.y; + const centre = { x: bounds.x + bounds.w / 2, y: bounds.y + bounds.h / 2 }; + rotateBy(e, centre, { x: centre.x, y: bounds.y - 24 }, 50, true); + e.selection.set([a]); + assert.equal(e.panel.value('Rotation'), '60°'); + e.selection.set([b]); + assert.equal(e.panel.value('Rotation'), '45°'); +}); + +test('a snapped rotation is one step to undo', () => { + const { e } = square(); + rotateBy(e, C, HANDLE, 50, true); + assert.equal(e.history.size, 1); + e.history.undo(); + assert.equal(e.panel.value('Rotation'), '0°'); +}); diff --git a/benchmarks/proof-build/hidden/B-S2/constraints.test.ts b/benchmarks/proof-build/hidden/B-S2/constraints.test.ts new file mode 100644 index 00000000..0be579be --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S2/constraints.test.ts @@ -0,0 +1,38 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function shiftRotate(degrees: number) { + const e = createEditor(); + const id = e.create('rect', { x: 0, y: 0, width: 100, height: 100 }); + e.selection.set([id]); + const start = -Math.PI / 2; + const end = start + (degrees * Math.PI) / 180; + e.canvas.drag( + { x: 50, y: -24 }, + { x: 50 + 74 * Math.cos(end), y: 50 + 74 * Math.sin(end) }, + { shift: true } + ); + return { e, id }; +} + +test('rotation is stored in radians, whatever Shift does', () => { + for (const degrees of [50, 100, -37, 170]) { + const { e, id } = shiftRotate(degrees); + const r = e.doc.require(id).rotation; + assert.ok(r > -Math.PI - 1e-9 && r <= Math.PI + 1e-9, `rotation ${r}`); + const shown = Number(e.panel.value('Rotation')!.replace('°', '')); + assert.ok( + Math.abs((r * 180) / Math.PI - shown) < 0.05, + `${r} shows ${shown}` + ); + } +}); + +test('a snapped angle is stored as the exact radian value', () => { + const { e, id } = shiftRotate(50); + if (e.panel.value('Rotation') !== '45°') return; + assert.ok(Math.abs(e.doc.require(id).rotation - Math.PI / 4) < 1e-9); + const saved = JSON.parse(e.files.save()); + assert.ok(Math.abs(saved.nodes[id].rotation - Math.PI / 4) < 1e-9); +}); diff --git a/benchmarks/proof-build/hidden/B-S2/naive.patch b/benchmarks/proof-build/hidden/B-S2/naive.patch new file mode 100644 index 00000000..a227be80 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S2/naive.patch @@ -0,0 +1,64 @@ +diff --git a/src/canvas/gestures.ts b/src/canvas/gestures.ts +index 13b0785..7a2080f 100644 +--- a/src/canvas/gestures.ts ++++ b/src/canvas/gestures.ts +@@ -9,6 +9,18 @@ import { hitTarget } from '../selection/hit.ts'; + import { selectionBounds } from '../selection/bounds.ts'; + import type { Viewport } from './viewport.ts'; + import { handleAt } from './handles.ts'; ++import { radToDeg } from '../util/math.ts'; ++ ++/** With Shift, the rotation handle snaps to this many degrees. */ ++export const ROTATE_SNAP_DEGREES = 15; ++ ++/** Maps an angle in degrees into (-180, 180]. */ ++function normalizeDegrees(degrees: number): number { ++ let a = degrees % 360; ++ if (a <= -180) a += 360; ++ if (a > 180) a -= 360; ++ return a; ++} + + export interface Modifiers { + shift?: boolean; +@@ -96,7 +108,7 @@ export class CanvasGestures { + this.drag = { kind: 'move', gesture: this.ctx.newGesture(), last: p }; + } + +- move(screen: Point, _mods: Modifiers = {}): void { ++ move(screen: Point, mods: Modifiers = {}): void { + const p = this.viewport.toDocument(screen); + const drag = this.drag; + if (drag.kind === 'move') { +@@ -122,18 +134,19 @@ export class CanvasGestures { + ); + } else if (drag.kind === 'rotate') { + const delta = angleAround(drag.centre, p) - drag.startAngle; +- for (const [id, start] of drag.from) +- this.ctx.dispatch( +- command('set-prop', { +- id, +- key: 'rotation', +- value: normalizeAngle(start + delta), +- }), +- { +- label: 'Rotate', +- gesture: drag.gesture, +- } +- ); ++ for (const [id, start] of drag.from) { ++ // Rotation is stored in degrees; the pointer angle is in radians. ++ const value = mods.shift ++ ? normalizeDegrees( ++ Math.round((start + radToDeg(delta)) / ROTATE_SNAP_DEGREES) * ++ ROTATE_SNAP_DEGREES ++ ) ++ : normalizeAngle(start + delta); ++ this.ctx.dispatch(command('set-prop', { id, key: 'rotation', value }), { ++ label: 'Rotate', ++ gesture: drag.gesture, ++ }); ++ } + } + } + diff --git a/benchmarks/proof-build/hidden/B-S2/reference.patch b/benchmarks/proof-build/hidden/B-S2/reference.patch new file mode 100644 index 00000000..039ac068 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S2/reference.patch @@ -0,0 +1,52 @@ +diff --git a/src/canvas/gestures.ts b/src/canvas/gestures.ts +index 13b0785..4bef122 100644 +--- a/src/canvas/gestures.ts ++++ b/src/canvas/gestures.ts +@@ -10,6 +10,9 @@ import { selectionBounds } from '../selection/bounds.ts'; + import type { Viewport } from './viewport.ts'; + import { handleAt } from './handles.ts'; + ++/** With Shift, the rotation handle snaps each layer's angle to this step (15°). */ ++export const ROTATE_SNAP = Math.PI / 12; ++ + export interface Modifiers { + shift?: boolean; + alt?: boolean; +@@ -96,7 +99,7 @@ export class CanvasGestures { + this.drag = { kind: 'move', gesture: this.ctx.newGesture(), last: p }; + } + +- move(screen: Point, _mods: Modifiers = {}): void { ++ move(screen: Point, mods: Modifiers = {}): void { + const p = this.viewport.toDocument(screen); + const drag = this.drag; + if (drag.kind === 'move') { +@@ -122,18 +125,16 @@ export class CanvasGestures { + ); + } else if (drag.kind === 'rotate') { + const delta = angleAround(drag.centre, p) - drag.startAngle; +- for (const [id, start] of drag.from) +- this.ctx.dispatch( +- command('set-prop', { +- id, +- key: 'rotation', +- value: normalizeAngle(start + delta), +- }), +- { +- label: 'Rotate', +- gesture: drag.gesture, +- } +- ); ++ for (const [id, start] of drag.from) { ++ const free = normalizeAngle(start + delta); ++ const value = mods.shift ++ ? normalizeAngle(Math.round(free / ROTATE_SNAP) * ROTATE_SNAP) ++ : free; ++ this.ctx.dispatch(command('set-prop', { id, key: 'rotation', value }), { ++ label: 'Rotate', ++ gesture: drag.gesture, ++ }); ++ } + } + } + diff --git a/benchmarks/proof-build/hidden/B-S3/acceptance.test.ts b/benchmarks/proof-build/hidden/B-S3/acceptance.test.ts new file mode 100644 index 00000000..d7b3ce7d --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S3/acceptance.test.ts @@ -0,0 +1,63 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const a = e.create('rect', { x: 0 }); + const b = e.create('rect', { x: 100 }); + const c = e.create('rect', { x: 200 }); + return { e, a, b, c }; +} +const shown = (e: ReturnType, ids: string[]) => + ids.map((id) => e.doc.require(id).visible); + +test('Mod+Shift+H hides the selected layers, and again shows them', () => { + const { e, a, b, c } = setup(); + e.selection.set([a, b]); + assert.ok(e.keys.press('Mod+Shift+H')); + assert.deepEqual(shown(e, [a, b, c]), [false, false, true]); + e.keys.press('Mod+Shift+H'); + assert.deepEqual(shown(e, [a, b, c]), [true, true, true]); +}); + +test('when only some are hidden it hides them all', () => { + const { e, a, b } = setup(); + e.selection.set([a]); + e.keys.press('Mod+Shift+H'); + e.selection.set([a, b]); + e.keys.press('Mod+Shift+H'); + assert.deepEqual(shown(e, [a, b]), [false, false]); +}); + +test('it is one step to undo, labelled Hide or Show', () => { + const { e, a, b } = setup(); + e.selection.set([a, b]); + e.keys.press('Mod+Shift+H'); + assert.equal(e.history.size, 1); + assert.equal(e.history.entries().at(-1)?.label, 'Hide'); + e.keys.press('Mod+Shift+H'); + assert.equal(e.history.entries().at(-1)?.label, 'Show'); + e.history.undo(); + assert.deepEqual(shown(e, [a, b]), [false, false]); + e.history.undo(); + assert.deepEqual(shown(e, [a, b]), [true, true]); +}); + +test('any spelling of the chord works', () => { + const { e, a } = setup(); + e.selection.set([a]); + e.keys.press('shift+cmd+h'); + assert.equal(e.doc.require(a).visible, false); +}); + +test('it is a canvas shortcut and does nothing with no selection', () => { + const { e, a } = setup(); + assert.equal(e.history.size, 0); + e.keys.press('Mod+Shift+H'); + assert.equal(e.history.size, 0); + e.selection.set([a]); + e.keys.focus('panel'); + e.keys.press('Mod+Shift+H'); + assert.equal(e.doc.require(a).visible, true); +}); diff --git a/benchmarks/proof-build/hidden/B-S3/constraints.test.ts b/benchmarks/proof-build/hidden/B-S3/constraints.test.ts new file mode 100644 index 00000000..cc285e77 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S3/constraints.test.ts @@ -0,0 +1,13 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +test('the shortcut table plugins read stays the frozen v1 table', () => { + const e = createEditor(); + let api: any; + e.plugins.load({ name: 'probe', main: (a) => (api = a) }); + const table = api.shortcuts(); + assert.equal(table.canvas['Mod+Shift+H'], undefined); + assert.equal(Object.keys(table.canvas).length, 18); + assert.ok(Object.isFrozen(table) && Object.isFrozen(table.canvas)); +}); diff --git a/benchmarks/proof-build/hidden/B-S3/naive.patch b/benchmarks/proof-build/hidden/B-S3/naive.patch new file mode 100644 index 00000000..c14007f4 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S3/naive.patch @@ -0,0 +1,43 @@ +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..c4e1d4d 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -3,6 +3,7 @@ import { command } from '../../commands/types.ts'; + import type { Direction } from '../../commands/builtin/nudge.ts'; + import { nudgeDelta } from '../../commands/builtin/nudge.ts'; + import { lockItems } from '../../commands/builtin/lock.ts'; ++import { visibilityItems } from '../../commands/builtin/visibility.ts'; + + function palette(chord: string, title: string): Binding { + return { chord, run: (ctx) => void ctx.palette.run(title) }; +@@ -40,6 +41,18 @@ export const canvasBindings: readonly Binding[] = [ + }); + }, + }, ++ { ++ chord: 'Mod+Shift+H', ++ run(ctx) { ++ const ids = ctx.selection.ids(); ++ if (!ids.length) return; ++ const show = ids.every((id) => !ctx.doc.require(id).visible); ++ ctx.dispatch( ++ command('set-visibility', { items: visibilityItems(ids, show) }), ++ { label: show ? 'Show' : 'Hide' } ++ ); ++ }, ++ }, + nudge('ArrowLeft', 'left', false), + nudge('ArrowRight', 'right', false), + nudge('ArrowUp', 'up', false), +diff --git a/src/keys/keymap.ts b/src/keys/keymap.ts +index 6412152..9904eab 100644 +--- a/src/keys/keymap.ts ++++ b/src/keys/keymap.ts +@@ -29,6 +29,7 @@ export const DEFAULT_KEYMAP: Keymap = deepFreeze({ + 'Mod+Shift+]': 'bring-to-front', + 'Mod+Shift+[': 'send-to-back', + 'Mod+Shift+L': 'toggle-lock', ++ 'Mod+Shift+H': 'toggle-visibility', + ArrowLeft: 'nudge-left', + ArrowRight: 'nudge-right', + ArrowUp: 'nudge-up', diff --git a/benchmarks/proof-build/hidden/B-S3/reference.patch b/benchmarks/proof-build/hidden/B-S3/reference.patch new file mode 100644 index 00000000..d05ec3d8 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S3/reference.patch @@ -0,0 +1,31 @@ +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..c4e1d4d 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -3,6 +3,7 @@ import { command } from '../../commands/types.ts'; + import type { Direction } from '../../commands/builtin/nudge.ts'; + import { nudgeDelta } from '../../commands/builtin/nudge.ts'; + import { lockItems } from '../../commands/builtin/lock.ts'; ++import { visibilityItems } from '../../commands/builtin/visibility.ts'; + + function palette(chord: string, title: string): Binding { + return { chord, run: (ctx) => void ctx.palette.run(title) }; +@@ -40,6 +41,18 @@ export const canvasBindings: readonly Binding[] = [ + }); + }, + }, ++ { ++ chord: 'Mod+Shift+H', ++ run(ctx) { ++ const ids = ctx.selection.ids(); ++ if (!ids.length) return; ++ const show = ids.every((id) => !ctx.doc.require(id).visible); ++ ctx.dispatch( ++ command('set-visibility', { items: visibilityItems(ids, show) }), ++ { label: show ? 'Show' : 'Hide' } ++ ); ++ }, ++ }, + nudge('ArrowLeft', 'left', false), + nudge('ArrowRight', 'right', false), + nudge('ArrowUp', 'up', false), diff --git a/benchmarks/proof-build/hidden/B-S4/acceptance.test.ts b/benchmarks/proof-build/hidden/B-S4/acceptance.test.ts new file mode 100644 index 00000000..bdc6e7e9 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S4/acceptance.test.ts @@ -0,0 +1,88 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const f = e.create('frame', { x: 0, y: 0, width: 400, height: 300 }); + const inFrame = e.create( + 'rect', + { x: 10, y: 10, width: 50, height: 50 }, + { parent: f } + ); + const g = e.create('group', {}, { parent: f }); + const inGroup = e.create( + 'rect', + { x: 100, y: 100, width: 20, height: 20 }, + { parent: g } + ); + const loose = e.create('rect', { x: 500, y: 0, width: 50, height: 50 }); + return { e, f, inFrame, inGroup, loose }; +} + +const size = (e: ReturnType, id: string) => [ + e.doc.require(id).width, + e.doc.require(id).height, +]; + +test('W and H take inches, points and px', () => { + const { e, loose } = setup(); + e.selection.set([loose]); + assert.ok(e.panel.edit('W', '2in').ok); + assert.ok(e.panel.edit('H', '18pt').ok); + assert.deepEqual(size(e, loose), [192, 24]); + e.panel.edit('W', '10px'); + e.panel.edit('H', '0.5 in'); + assert.deepEqual(size(e, loose), [10, 48]); +}); + +test('arithmetic still works and can mix units', () => { + const { e, loose } = setup(); + e.selection.set([loose]); + e.panel.edit('W', '1in + 4'); + assert.equal(e.doc.require(loose).width, 100); + e.panel.edit('W', '(1in - 16) / 2'); + assert.equal(e.doc.require(loose).width, 40); + e.panel.edit('H', '72pt - 1in + 30'); + assert.equal(e.doc.require(loose).height, 30); + e.panel.edit('W', '50 * 2 + 10'); + assert.equal(e.doc.require(loose).width, 110); +}); + +test('% is a share of the parent frame: its width for W, its height for H', () => { + const { e, inFrame } = setup(); + e.selection.set([inFrame]); + e.panel.edit('W', '50%'); + e.panel.edit('H', '10%'); + assert.deepEqual(size(e, inFrame), [200, 30]); + e.panel.edit('W', '25% + 1in'); + assert.equal(e.doc.require(inFrame).width, 196); +}); + +test('a layer in a group inside a frame takes the frame', () => { + const { e, inGroup } = setup(); + e.selection.set([inGroup]); + e.panel.edit('W', '10%'); + assert.equal(e.doc.require(inGroup).width, 40); +}); + +test('% on a layer that is not inside a frame is refused', () => { + const { e, loose } = setup(); + e.selection.set([loose]); + const result = e.panel.edit('W', '50%'); + assert.equal(result.ok, false); + assert.deepEqual(size(e, loose), [50, 50]); + assert.equal(e.history.size, 0); +}); + +test('other fields work as before', () => { + const { e, loose } = setup(); + e.selection.set([loose]); + e.panel.edit('X', '12 + 3'); + assert.equal(e.doc.require(loose).x, 15); + e.panel.edit('Opacity', '40%'); + assert.equal(e.doc.require(loose).opacity, 0.4); + e.panel.edit('Rotation', '90°'); + assert.equal(e.panel.value('Rotation'), '90°'); + assert.equal(e.panel.edit('W', 'wide').ok, false); +}); diff --git a/benchmarks/proof-build/hidden/B-S4/constraints.test.ts b/benchmarks/proof-build/hidden/B-S4/constraints.test.ts new file mode 100644 index 00000000..4a435b6f --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S4/constraints.test.ts @@ -0,0 +1,49 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; +import { parseFile } from '../../src/persist/deserialize.ts'; + +/** A version 1 file with length strings in units, and the sizes it opened with before. */ +const LEGACY = JSON.stringify({ + version: 1, + name: 'Legacy units', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Sheet', + parent: null, + children: ['rect-2'], + x: '0px', + y: '0', + width: '8.5in', + height: '11in', + fill: '#FFFFFF', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Margin', + parent: 'frame-1', + x: '50%', + y: '18pt', + width: '2in', + height: ' 12 px ', + }, + ], +}); + +test('old files with unit strings open as they always did', () => { + const data = parseFile(LEGACY); + const sheet = data.nodes['frame-1']; + const margin = data.nodes['rect-2']; + assert.deepEqual([sheet.width, sheet.height], [0, 0]); + assert.deepEqual( + [margin.x, margin.y, margin.width, margin.height], + [0, 0, 0, 12] + ); + const e = createEditor(); + e.files.load(LEGACY); + e.selection.set(['rect-2']); + assert.equal(e.panel.value('H'), '12'); +}); diff --git a/benchmarks/proof-build/hidden/B-S4/naive.patch b/benchmarks/proof-build/hidden/B-S4/naive.patch new file mode 100644 index 00000000..1794aab7 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S4/naive.patch @@ -0,0 +1,196 @@ +diff --git a/src/panel/commit.ts b/src/panel/commit.ts +index c24ca8c..3ec552d 100644 +--- a/src/panel/commit.ts ++++ b/src/panel/commit.ts +@@ -1,12 +1,16 @@ + import type { EditorContext } from '../commands/types.ts'; +-import type { FieldDescriptor } from './schema.ts'; ++import type { FieldDescriptor, FieldValue } from './schema.ts'; + import { clampField, parseFieldInput } from './fields.ts'; + + export type CommitResult = + | { ok: true; changed: number } + | { ok: false; error: string }; + +-/** Commits typed text to one field for every selected node it applies to. */ ++/** ++ * Commits typed text to one field for every selected node it applies to. ++ * A field that reads units reads the text once per node, since `%` depends ++ * on where the node sits; if any node cannot take it, nothing changes. ++ */ + export function commitField( + ctx: EditorContext, + field: FieldDescriptor, +@@ -14,10 +18,12 @@ export function commitField( + ): CommitResult { + if (field.readonly) + return { ok: false, error: `${field.label} is read-only` }; +- const parsed = parseFieldInput(field.kind, text); +- if (parsed === null) +- return { ok: false, error: `cannot read "${text}" as ${field.kind}` }; +- const value = clampField(field, parsed); ++ const refused: CommitResult = { ++ ok: false, ++ error: `cannot read "${text}" as ${field.kind}`, ++ }; ++ if (!field.units && parseFieldInput(field.kind, text) === null) ++ return refused; + const nodes = ctx.selection + .ids() + .map((id) => ctx.doc.require(id)) +@@ -27,10 +33,20 @@ export function commitField( + ok: false, + error: `${field.label} does not apply to the selection`, + }; ++ const values = new Map(); ++ for (const node of nodes) { ++ const parsed = parseFieldInput( ++ field.kind, ++ text, ++ field.units?.(node, ctx.doc) ++ ); ++ if (parsed === null) return refused; ++ values.set(node.id, clampField(field, parsed)); ++ } + const gesture = ctx.newGesture(); + let changed = 0; + for (const node of nodes) +- for (const c of field.write(node, value, ctx.doc)) { ++ for (const c of field.write(node, values.get(node.id)!, ctx.doc)) { + ctx.dispatch(c, { label: `Set ${field.label}`, gesture }); + changed += 1; + } +diff --git a/src/panel/fields.ts b/src/panel/fields.ts +index 2643eef..7d7d15f 100644 +--- a/src/panel/fields.ts ++++ b/src/panel/fields.ts +@@ -1,5 +1,7 @@ + import type { FieldDescriptor, FieldKind, FieldValue } from './schema.ts'; + import { parseExpression } from '../units/expr.ts'; ++import { parseLength } from '../units/parse.ts'; ++import type { UnitContext } from '../units/registry.ts'; + import { displayNumber } from '../units/precision.ts'; + import { formatAngle, formatLength, formatPercent } from '../units/format.ts'; + import { formatColor, parseColor } from '../model/style.ts'; +@@ -8,10 +10,12 @@ import { clamp, degToRad } from '../util/math.ts'; + /** Reads what the user typed into a field. Returns null when it cannot. */ + export function parseFieldInput( + kind: FieldKind, +- text: string ++ text: string, ++ units?: UnitContext + ): FieldValue | null { + switch (kind) { + case 'length': ++ return units ? toLength(text, units) : parseExpression(text); + case 'number': + return parseExpression(text); + case 'angle': { +@@ -46,6 +50,11 @@ export function formatFieldValue(kind: FieldKind, value: FieldValue): string { + } + } + ++/** A typed length: a number with a unit through parseLength, else an expression. */ ++export function toLength(text: string, units: UnitContext): number | null { ++ return parseLength(text, units) ?? parseExpression(text); ++} ++ + /** Applies the field's bounds, if it has any. */ + export function clampField( + field: FieldDescriptor, +diff --git a/src/panel/schema.ts b/src/panel/schema.ts +index 397b4d5..9d40c65 100644 +--- a/src/panel/schema.ts ++++ b/src/panel/schema.ts +@@ -1,6 +1,7 @@ + import type { Document } from '../model/document.ts'; + import type { SceneNode } from '../model/types.ts'; + import type { Command } from '../commands/types.ts'; ++import type { UnitContext } from '../units/registry.ts'; + + /** + * How a field reads and writes its value: +@@ -29,6 +30,11 @@ export interface FieldDescriptor { + min?: number; + max?: number; + appliesTo(node: SceneNode): boolean; ++ /** ++ * For a length field that reads units (in, pt, px, %): what `%` is ++ * relative to for `node`. Typed text is then read once per node. ++ */ ++ units?(node: SceneNode, doc: Document): UnitContext; + read(node: SceneNode, doc: Document): FieldValue; + /** The commands that give `node` this value. */ + write(node: SceneNode, value: FieldValue, doc: Document): Command[]; +diff --git a/src/panel/sections/layout.ts b/src/panel/sections/layout.ts +index ca071d4..7e5925c 100644 +--- a/src/panel/sections/layout.ts ++++ b/src/panel/sections/layout.ts +@@ -6,9 +6,19 @@ import { scaledSubtree } from '../../commands/builtin/resize.ts'; + import { rectOf } from '../../model/geometry.ts'; + import type { SceneNode } from '../../model/types.ts'; + import { normalizeAngle } from '../../model/transform.ts'; ++import type { Document } from '../../model/document.ts'; ++import { ancestors } from '../../model/tree.ts'; + + const any = () => true; + ++/** What `%` in W or H means: a share of the nearest frame around the node. */ ++function frameShare(axis: 'width' | 'height') { ++ return (node: SceneNode, doc: Document) => { ++ const frame = ancestors(doc, node.id).find((a) => a.type === 'frame'); ++ return frame ? { reference: frame[axis] } : {}; ++ }; ++} ++ + function moveTo(node: SceneNode, axis: 'x' | 'y', value: number) { + const delta = value - node[axis]; + if (delta === 0) return []; +@@ -44,6 +54,7 @@ export const layoutSection: SectionDescriptor = { + kind: 'length', + min: 1, + appliesTo: any, ++ units: frameShare('width'), + read: (node) => node.width, + write: (node, value, doc) => [ + command('resize', { +@@ -59,6 +70,7 @@ export const layoutSection: SectionDescriptor = { + kind: 'length', + min: 1, + appliesTo: (node) => node.type !== 'line', ++ units: frameShare('height'), + read: (node) => node.height, + write: (node, value, doc) => [ + command('resize', { +diff --git a/src/units/parse.ts b/src/units/parse.ts +index cda1196..e25c13f 100644 +--- a/src/units/parse.ts ++++ b/src/units/parse.ts +@@ -1,13 +1,23 @@ + // Reads a length written as a string, such as `12`, `12px` or ` 4.5 px `. + // Version 1 files stored lengths this way; the v1→v2 migration reads them +-// through here. ++// through here. Panel fields read typed lengths through here too, with a ++// unit context, so `2in`, `18pt` and `50%` are read as px. ++ ++import type { UnitContext, UnitId } from './registry.ts'; ++import { toPx, unitFor } from './registry.ts'; + + const LENGTH = /^\s*(-?(?:\d+(?:\.\d*)?|\.\d+))\s*(px)?\s*$/i; ++const LENGTH_WITH_UNIT = /^\s*(-?(?:\d+(?:\.\d*)?|\.\d+))\s*(px|pt|in|%)?\s*$/i; + +-export function parseLength(text: string): number | null { +- const match = LENGTH.exec(text); ++export function parseLength(text: string, units?: UnitContext): number | null { ++ const match = (units ? LENGTH_WITH_UNIT : LENGTH).exec(text); + if (!match) return null; +- const value = Number(match[1]); ++ let value = Number(match[1]); ++ if (units && match[2]) { ++ const px = toPx(value, unitFor(match[2])!.id as UnitId, units); ++ if (px === null) return null; ++ value = px; ++ } + return Number.isFinite(value) ? value : null; + } + diff --git a/benchmarks/proof-build/hidden/B-S4/reference.patch b/benchmarks/proof-build/hidden/B-S4/reference.patch new file mode 100644 index 00000000..eb8db549 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-S4/reference.patch @@ -0,0 +1,220 @@ +diff --git a/src/panel/commit.ts b/src/panel/commit.ts +index c24ca8c..3ec552d 100644 +--- a/src/panel/commit.ts ++++ b/src/panel/commit.ts +@@ -1,12 +1,16 @@ + import type { EditorContext } from '../commands/types.ts'; +-import type { FieldDescriptor } from './schema.ts'; ++import type { FieldDescriptor, FieldValue } from './schema.ts'; + import { clampField, parseFieldInput } from './fields.ts'; + + export type CommitResult = + | { ok: true; changed: number } + | { ok: false; error: string }; + +-/** Commits typed text to one field for every selected node it applies to. */ ++/** ++ * Commits typed text to one field for every selected node it applies to. ++ * A field that reads units reads the text once per node, since `%` depends ++ * on where the node sits; if any node cannot take it, nothing changes. ++ */ + export function commitField( + ctx: EditorContext, + field: FieldDescriptor, +@@ -14,10 +18,12 @@ export function commitField( + ): CommitResult { + if (field.readonly) + return { ok: false, error: `${field.label} is read-only` }; +- const parsed = parseFieldInput(field.kind, text); +- if (parsed === null) +- return { ok: false, error: `cannot read "${text}" as ${field.kind}` }; +- const value = clampField(field, parsed); ++ const refused: CommitResult = { ++ ok: false, ++ error: `cannot read "${text}" as ${field.kind}`, ++ }; ++ if (!field.units && parseFieldInput(field.kind, text) === null) ++ return refused; + const nodes = ctx.selection + .ids() + .map((id) => ctx.doc.require(id)) +@@ -27,10 +33,20 @@ export function commitField( + ok: false, + error: `${field.label} does not apply to the selection`, + }; ++ const values = new Map(); ++ for (const node of nodes) { ++ const parsed = parseFieldInput( ++ field.kind, ++ text, ++ field.units?.(node, ctx.doc) ++ ); ++ if (parsed === null) return refused; ++ values.set(node.id, clampField(field, parsed)); ++ } + const gesture = ctx.newGesture(); + let changed = 0; + for (const node of nodes) +- for (const c of field.write(node, value, ctx.doc)) { ++ for (const c of field.write(node, values.get(node.id)!, ctx.doc)) { + ctx.dispatch(c, { label: `Set ${field.label}`, gesture }); + changed += 1; + } +diff --git a/src/panel/fields.ts b/src/panel/fields.ts +index 2643eef..6cba59f 100644 +--- a/src/panel/fields.ts ++++ b/src/panel/fields.ts +@@ -1,5 +1,6 @@ + import type { FieldDescriptor, FieldKind, FieldValue } from './schema.ts'; +-import { parseExpression } from '../units/expr.ts'; ++import { parseExpression, parseLengthExpression } from '../units/expr.ts'; ++import type { UnitContext } from '../units/registry.ts'; + import { displayNumber } from '../units/precision.ts'; + import { formatAngle, formatLength, formatPercent } from '../units/format.ts'; + import { formatColor, parseColor } from '../model/style.ts'; +@@ -8,10 +9,12 @@ import { clamp, degToRad } from '../util/math.ts'; + /** Reads what the user typed into a field. Returns null when it cannot. */ + export function parseFieldInput( + kind: FieldKind, +- text: string ++ text: string, ++ units?: UnitContext + ): FieldValue | null { + switch (kind) { + case 'length': ++ return units ? parseLengthExpression(text, units) : parseExpression(text); + case 'number': + return parseExpression(text); + case 'angle': { +diff --git a/src/panel/schema.ts b/src/panel/schema.ts +index 397b4d5..9d40c65 100644 +--- a/src/panel/schema.ts ++++ b/src/panel/schema.ts +@@ -1,6 +1,7 @@ + import type { Document } from '../model/document.ts'; + import type { SceneNode } from '../model/types.ts'; + import type { Command } from '../commands/types.ts'; ++import type { UnitContext } from '../units/registry.ts'; + + /** + * How a field reads and writes its value: +@@ -29,6 +30,11 @@ export interface FieldDescriptor { + min?: number; + max?: number; + appliesTo(node: SceneNode): boolean; ++ /** ++ * For a length field that reads units (in, pt, px, %): what `%` is ++ * relative to for `node`. Typed text is then read once per node. ++ */ ++ units?(node: SceneNode, doc: Document): UnitContext; + read(node: SceneNode, doc: Document): FieldValue; + /** The commands that give `node` this value. */ + write(node: SceneNode, value: FieldValue, doc: Document): Command[]; +diff --git a/src/panel/sections/layout.ts b/src/panel/sections/layout.ts +index ca071d4..7e5925c 100644 +--- a/src/panel/sections/layout.ts ++++ b/src/panel/sections/layout.ts +@@ -6,9 +6,19 @@ import { scaledSubtree } from '../../commands/builtin/resize.ts'; + import { rectOf } from '../../model/geometry.ts'; + import type { SceneNode } from '../../model/types.ts'; + import { normalizeAngle } from '../../model/transform.ts'; ++import type { Document } from '../../model/document.ts'; ++import { ancestors } from '../../model/tree.ts'; + + const any = () => true; + ++/** What `%` in W or H means: a share of the nearest frame around the node. */ ++function frameShare(axis: 'width' | 'height') { ++ return (node: SceneNode, doc: Document) => { ++ const frame = ancestors(doc, node.id).find((a) => a.type === 'frame'); ++ return frame ? { reference: frame[axis] } : {}; ++ }; ++} ++ + function moveTo(node: SceneNode, axis: 'x' | 'y', value: number) { + const delta = value - node[axis]; + if (delta === 0) return []; +@@ -44,6 +54,7 @@ export const layoutSection: SectionDescriptor = { + kind: 'length', + min: 1, + appliesTo: any, ++ units: frameShare('width'), + read: (node) => node.width, + write: (node, value, doc) => [ + command('resize', { +@@ -59,6 +70,7 @@ export const layoutSection: SectionDescriptor = { + kind: 'length', + min: 1, + appliesTo: (node) => node.type !== 'line', ++ units: frameShare('height'), + read: (node) => node.height, + write: (node, value, doc) => [ + command('resize', { +diff --git a/src/units/expr.ts b/src/units/expr.ts +index e1e055f..e4e7973 100644 +--- a/src/units/expr.ts ++++ b/src/units/expr.ts +@@ -1,6 +1,10 @@ + // The expression parser behind every numeric panel field. It reads + // arithmetic on plain numbers: `12`, `12px`, `100 / 3`, `(8 + 4) * 2`, + // `-5`. A `px` suffix is allowed on any number and means nothing extra. ++// parseLengthExpression also reads `in`, `pt` and `%` on each number. ++ ++import type { UnitContext, UnitId } from './registry.ts'; ++import { toPx, unitFor } from './registry.ts'; + + type Token = + | { kind: 'num'; value: number } +@@ -8,8 +12,10 @@ type Token = + | { kind: 'paren'; value: '(' | ')' }; + + const NUMBER = /^(\d+(?:\.\d*)?|\.\d+)(?:\s*px)?/i; ++const NUMBER_WITH_UNIT = /^(\d+(?:\.\d*)?|\.\d+)(?:\s*(px|pt|in|%))?/i; + +-function tokenize(text: string): Token[] | null { ++/** With `units`, a number may carry a unit and is read as px. */ ++function tokenize(text: string, units?: UnitContext): Token[] | null { + const tokens: Token[] = []; + let rest = text.trim(); + while (rest.length) { +@@ -28,9 +34,15 @@ function tokenize(text: string): Token[] | null { + rest = rest.slice(1); + continue; + } +- const match = NUMBER.exec(rest); ++ const match = (units ? NUMBER_WITH_UNIT : NUMBER).exec(rest); + if (!match) return null; +- tokens.push({ kind: 'num', value: Number(match[1]) }); ++ let value = Number(match[1]); ++ if (units && match[2]) { ++ const px = toPx(value, unitFor(match[2])!.id as UnitId, units); ++ if (px === null) return null; ++ value = px; ++ } ++ tokens.push({ kind: 'num', value }); + rest = rest.slice(match[0].length); + } + return tokens; +@@ -117,7 +129,21 @@ class Parser { + + /** Evaluates a typed field value. Returns null for anything it cannot read. */ + export function parseExpression(text: string): number | null { +- const tokens = tokenize(text); ++ return evaluate(tokenize(text)); ++} ++ ++/** ++ * Evaluates a typed length in px. Each number may carry a unit (`2in`, ++ * `18pt`, `50%`); `%` needs `context.reference` and is refused without one. ++ */ ++export function parseLengthExpression( ++ text: string, ++ context: UnitContext = {} ++): number | null { ++ return evaluate(tokenize(text, context)); ++} ++ ++function evaluate(tokens: Token[] | null): number | null { + if (!tokens || !tokens.length) return null; + const value = new Parser(tokens).parse(); + return value !== null && Number.isFinite(value) ? value : null; diff --git a/benchmarks/proof-build/hidden/B-V1/acceptance.test.ts b/benchmarks/proof-build/hidden/B-V1/acceptance.test.ts new file mode 100644 index 00000000..d6acbb75 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V1/acceptance.test.ts @@ -0,0 +1,119 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const sel = (e: Editor) => [...e.selection.ids()].sort(); +const sorted = (ids: string[]) => [...ids].sort(); + +function row() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0, y: 0, width: 50, height: 50 }); + const b = e.create('rect', { + name: 'B', + x: 100, + y: 0, + width: 50, + height: 50, + }); + const c = e.create('rect', { + name: 'C', + x: 200, + y: 0, + width: 50, + height: 50, + }); + return { e, a, b, c }; +} + +test('dragging on empty canvas selects every layer fully inside the box', () => { + const { e, a, b } = row(); + e.canvas.drag({ x: -10, y: -10 }, { x: 160, y: 60 }); + assert.deepEqual(sel(e), sorted([a, b])); +}); + +test('the box works whichever way it is drawn', () => { + const { e, b, c } = row(); + e.canvas.drag({ x: 260, y: 70 }, { x: 90, y: -5 }); + assert.deepEqual(sel(e), sorted([b, c])); +}); + +test('a layer only partly inside the box is not selected', () => { + const { e, a } = row(); + e.canvas.drag({ x: -10, y: -10 }, { x: 120, y: 60 }); + assert.deepEqual(sel(e), [a]); +}); + +test('a group the box covers is selected whole; one it partly covers gives up its covered layers', () => { + const e = createEditor(); + const g = e.create('group', { name: 'G' }); + const g1 = e.create( + 'rect', + { x: 0, y: 0, width: 40, height: 40 }, + { parent: g } + ); + const g2 = e.create( + 'rect', + { x: 60, y: 0, width: 40, height: 40 }, + { parent: g } + ); + e.canvas.drag({ x: -5, y: -5 }, { x: 110, y: 50 }); + assert.deepEqual(sel(e), [g]); + e.canvas.drag({ x: -5, y: -5 }, { x: 50, y: 50 }); + assert.deepEqual(sel(e), [g1]); + assert.ok(!e.selection.has(g2)); +}); + +test('inside a frame the box reaches the layers it covers', () => { + const e = createEditor(); + const f = e.create('frame', { x: 50, y: 50, width: 400, height: 300 }); + const r1 = e.create( + 'rect', + { x: 70, y: 70, width: 40, height: 40 }, + { parent: f } + ); + const r2 = e.create( + 'rect', + { x: 130, y: 70, width: 40, height: 40 }, + { parent: f } + ); + e.create('rect', { x: 350, y: 250, width: 40, height: 40 }, { parent: f }); + e.canvas.drag({ x: 0, y: 0 }, { x: 180, y: 120 }); + assert.deepEqual(sel(e), sorted([r1, r2])); + e.canvas.drag({ x: 0, y: 0 }, { x: 460, y: 360 }); + assert.deepEqual(sel(e), [f]); +}); + +test('Shift adds the boxed layers to the selection', () => { + const { e, a, c } = row(); + e.selection.set([c]); + e.canvas.drag({ x: -10, y: -10 }, { x: 60, y: 60 }, { shift: true }); + assert.deepEqual(sel(e), sorted([a, c])); +}); + +test('without Shift the box replaces the selection', () => { + const { e, a, c } = row(); + e.selection.set([c]); + e.canvas.drag({ x: -10, y: -10 }, { x: 60, y: 60 }); + assert.deepEqual(sel(e), [a]); +}); + +test('the box moves nothing and is not an edit', () => { + const { e, a, b, c } = row(); + e.canvas.drag({ x: -10, y: -10 }, { x: 260, y: 60 }); + assert.deepEqual(sel(e), sorted([a, b, c])); + assert.equal(e.history.size, 0); + assert.deepEqual( + [a, b, c].map((id) => e.doc.require(id).x), + [0, 100, 200] + ); +}); + +test('a click on empty canvas still clears, and dragging a layer still moves it', () => { + const { e, a, b } = row(); + e.selection.set([a, b]); + e.canvas.click({ x: 500, y: 500 }); + assert.deepEqual(sel(e), []); + e.canvas.drag({ x: 10, y: 10 }, { x: 30, y: 10 }); + assert.equal(e.doc.require(a).x, 20); +}); diff --git a/benchmarks/proof-build/hidden/B-V1/constraints.test.ts b/benchmarks/proof-build/hidden/B-V1/constraints.test.ts new file mode 100644 index 00000000..3586ca9c --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V1/constraints.test.ts @@ -0,0 +1,57 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +const sel = (e: ReturnType) => + [...e.selection.ids()].sort(); + +test('hidden and locked layers are never selected by the box', () => { + const e = createEditor(); + const a = e.create('rect', { x: 0, y: 0, width: 40, height: 40 }); + e.create('rect', { x: 50, y: 0, width: 40, height: 40, visible: false }); + e.create('rect', { x: 100, y: 0, width: 40, height: 40, locked: true }); + e.canvas.drag({ x: -10, y: -10 }, { x: 200, y: 50 }); + assert.ok( + sel(e).every((id) => id === a), + String(sel(e)) + ); +}); + +test('nothing inside a hidden or locked container is selected by the box', () => { + const e = createEditor(); + const locked = e.create('frame', { + x: 0, + y: 0, + width: 300, + height: 300, + locked: true, + }); + e.create('rect', { x: 20, y: 20, width: 40, height: 40 }, { parent: locked }); + const hidden = e.create('group', { visible: false }); + e.create( + 'rect', + { x: 400, y: 20, width: 40, height: 40 }, + { parent: hidden } + ); + const shown = e.create('group'); + const inShown = e.create( + 'rect', + { x: 400, y: 200, width: 40, height: 40 }, + { parent: shown } + ); + e.create( + 'rect', + { x: 460, y: 200, width: 40, height: 40, locked: true }, + { parent: shown } + ); + e.canvas.drag({ x: 10, y: 10 }, { x: 450, y: 250 }); + assert.ok( + sel(e).every((id) => id === inShown), + String(sel(e)) + ); + e.canvas.drag({ x: -50, y: -50 }, { x: 600, y: 600 }); + assert.ok( + sel(e).every((id) => id === shown || id === inShown), + String(sel(e)) + ); +}); diff --git a/benchmarks/proof-build/hidden/B-V1/naive.patch b/benchmarks/proof-build/hidden/B-V1/naive.patch new file mode 100644 index 00000000..4f979af0 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V1/naive.patch @@ -0,0 +1,81 @@ +diff --git a/src/canvas/gestures.ts b/src/canvas/gestures.ts +index 13b0785..e2ca09b 100644 +--- a/src/canvas/gestures.ts ++++ b/src/canvas/gestures.ts +@@ -2,11 +2,13 @@ import type { EditorContext, Json } from '../commands/types.ts'; + import { command } from '../commands/types.ts'; + import { scaledSubtree } from '../commands/builtin/resize.ts'; + import type { Point, Rect } from '../model/geometry.ts'; +-import { centre, rectOf } from '../model/geometry.ts'; ++import { centre, normalizeRect, rectOf } from '../model/geometry.ts'; + import type { Handle } from '../model/transform.ts'; + import { angleAround, normalizeAngle, resizeRect } from '../model/transform.ts'; + import { hitTarget } from '../selection/hit.ts'; + import { selectionBounds } from '../selection/bounds.ts'; ++import { boxSelect } from '../selection/box.ts'; ++import { normalizeSelection } from '../selection/rules.ts'; + import type { Viewport } from './viewport.ts'; + import { handleAt } from './handles.ts'; + +@@ -18,6 +20,7 @@ export interface Modifiers { + + type Drag = + | { kind: 'none' } ++ | { kind: 'box'; start: Point; keep: string[] } + | { kind: 'move'; gesture: string; last: Point } + | { + kind: 'resize'; +@@ -84,7 +87,7 @@ export class CanvasGestures { + const target = hitTarget(doc, p); + if (target === null) { + if (!mods.shift) selection.clear(); +- this.drag = { kind: 'none' }; ++ this.drag = { kind: 'box', start: p, keep: selection.ids() }; + return; + } + if (mods.shift) { +@@ -138,6 +141,16 @@ export class CanvasGestures { + } + + up(screen: Point, mods: Modifiers = {}): void { ++ const drag = this.drag; ++ if (drag.kind === 'box') { ++ const { doc, selection } = this.ctx; ++ const box = normalizeRect(drag.start, this.viewport.toDocument(screen)); ++ selection.set( ++ normalizeSelection(doc, [...drag.keep, ...boxSelect(doc, box)]) ++ ); ++ this.drag = { kind: 'none' }; ++ return; ++ } + if (this.drag.kind !== 'none') this.move(screen, mods); + this.drag = { kind: 'none' }; + } +diff --git a/src/selection/box.ts b/src/selection/box.ts +new file mode 100644 +index 0000000..0691a38 +--- /dev/null ++++ b/src/selection/box.ts +@@ -0,0 +1,22 @@ ++import type { Document } from '../model/document.ts'; ++import type { Rect } from '../model/geometry.ts'; ++import { nodeBounds, rectContainsRect } from '../model/geometry.ts'; ++import { isContainer } from '../model/types.ts'; ++ ++/** ++ * What a selection box selects: every layer that lies fully inside `box`. ++ * A container the box covers is taken whole; for one it only partly covers, ++ * the box reaches the layers inside. ++ */ ++export function boxSelect(doc: Document, box: Rect): string[] { ++ const out: string[] = []; ++ const visit = (parent: string | null) => { ++ for (const id of doc.children(parent)) { ++ const node = doc.require(id); ++ if (rectContainsRect(box, nodeBounds(node))) out.push(id); ++ else if (isContainer(node)) visit(id); ++ } ++ }; ++ visit(null); ++ return out; ++} diff --git a/benchmarks/proof-build/hidden/B-V1/reference.patch b/benchmarks/proof-build/hidden/B-V1/reference.patch new file mode 100644 index 00000000..5a259659 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V1/reference.patch @@ -0,0 +1,84 @@ +diff --git a/src/canvas/gestures.ts b/src/canvas/gestures.ts +index 13b0785..e2ca09b 100644 +--- a/src/canvas/gestures.ts ++++ b/src/canvas/gestures.ts +@@ -2,11 +2,13 @@ import type { EditorContext, Json } from '../commands/types.ts'; + import { command } from '../commands/types.ts'; + import { scaledSubtree } from '../commands/builtin/resize.ts'; + import type { Point, Rect } from '../model/geometry.ts'; +-import { centre, rectOf } from '../model/geometry.ts'; ++import { centre, normalizeRect, rectOf } from '../model/geometry.ts'; + import type { Handle } from '../model/transform.ts'; + import { angleAround, normalizeAngle, resizeRect } from '../model/transform.ts'; + import { hitTarget } from '../selection/hit.ts'; + import { selectionBounds } from '../selection/bounds.ts'; ++import { boxSelect } from '../selection/box.ts'; ++import { normalizeSelection } from '../selection/rules.ts'; + import type { Viewport } from './viewport.ts'; + import { handleAt } from './handles.ts'; + +@@ -18,6 +20,7 @@ export interface Modifiers { + + type Drag = + | { kind: 'none' } ++ | { kind: 'box'; start: Point; keep: string[] } + | { kind: 'move'; gesture: string; last: Point } + | { + kind: 'resize'; +@@ -84,7 +87,7 @@ export class CanvasGestures { + const target = hitTarget(doc, p); + if (target === null) { + if (!mods.shift) selection.clear(); +- this.drag = { kind: 'none' }; ++ this.drag = { kind: 'box', start: p, keep: selection.ids() }; + return; + } + if (mods.shift) { +@@ -138,6 +141,16 @@ export class CanvasGestures { + } + + up(screen: Point, mods: Modifiers = {}): void { ++ const drag = this.drag; ++ if (drag.kind === 'box') { ++ const { doc, selection } = this.ctx; ++ const box = normalizeRect(drag.start, this.viewport.toDocument(screen)); ++ selection.set( ++ normalizeSelection(doc, [...drag.keep, ...boxSelect(doc, box)]) ++ ); ++ this.drag = { kind: 'none' }; ++ return; ++ } + if (this.drag.kind !== 'none') this.move(screen, mods); + this.drag = { kind: 'none' }; + } +diff --git a/src/selection/box.ts b/src/selection/box.ts +new file mode 100644 +index 0000000..30471cd +--- /dev/null ++++ b/src/selection/box.ts +@@ -0,0 +1,25 @@ ++import type { Document } from '../model/document.ts'; ++import type { Rect } from '../model/geometry.ts'; ++import { nodeBounds, rectContainsRect } from '../model/geometry.ts'; ++import { isContainer } from '../model/types.ts'; ++import { isInteractive } from './rules.ts'; ++ ++/** ++ * What a selection box selects: every layer that lies fully inside `box`. ++ * A container the box covers is taken whole; for one it only partly covers, ++ * the box reaches the layers inside. Like a click, it skips hidden and ++ * locked layers and everything inside them. ++ */ ++export function boxSelect(doc: Document, box: Rect): string[] { ++ const out: string[] = []; ++ const visit = (parent: string | null) => { ++ for (const id of doc.children(parent)) { ++ if (!isInteractive(doc, id)) continue; ++ const node = doc.require(id); ++ if (rectContainsRect(box, nodeBounds(node))) out.push(id); ++ else if (isContainer(node)) visit(id); ++ } ++ }; ++ visit(null); ++ return out; ++} diff --git a/benchmarks/proof-build/hidden/B-V2/acceptance.test.ts b/benchmarks/proof-build/hidden/B-V2/acceptance.test.ts new file mode 100644 index 00000000..c09ba72e --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V2/acceptance.test.ts @@ -0,0 +1,99 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const box = (e: Editor, id: string) => { + const n = e.doc.require(id); + return [n.x, n.y, n.width, n.height]; +}; + +function row() { + const e = createEditor(); + const a = e.create('rect', { x: 0, y: 0, width: 50, height: 50 }); + const b = e.create('rect', { x: 100, y: 0, width: 50, height: 50 }); + const c = e.create('rect', { x: 200, y: 0, width: 50, height: 50 }); + e.selection.set([a, b, c]); + return { e, a, b, c }; +} + +test('Scale selection scales sizes and positions about the centre of the selection', () => { + const { e, a, b, c } = row(); + assert.ok(e.palette.run('Scale selection', '200')); + assert.deepEqual(box(e, a), [-125, -25, 100, 100]); + assert.deepEqual(box(e, b), [75, -25, 100, 100]); + assert.deepEqual(box(e, c), [275, -25, 100, 100]); +}); + +test('a percent sign is fine, and smaller works too', () => { + const { e, a, c } = row(); + e.palette.run('Scale selection', '50%'); + assert.deepEqual(box(e, a), [62.5, 12.5, 25, 25]); + assert.deepEqual(box(e, c), [162.5, 12.5, 25, 25]); +}); + +test('a single layer scales about its own centre', () => { + const e = createEditor(); + const r = e.create('rect', { x: 10, y: 20, width: 40, height: 20 }); + e.selection.set([r]); + e.palette.run('Scale selection', '150'); + assert.deepEqual(box(e, r), [0, 15, 60, 30]); +}); + +test('what is inside a selected group or frame scales with it', () => { + const e = createEditor(); + const f = e.create('frame', { x: 0, y: 0, width: 100, height: 100 }); + const inner = e.create( + 'rect', + { x: 10, y: 10, width: 20, height: 20 }, + { parent: f } + ); + const g = e.create('group'); + const g1 = e.create( + 'rect', + { x: 200, y: 0, width: 20, height: 20 }, + { parent: g } + ); + const g2 = e.create( + 'rect', + { x: 280, y: 80, width: 20, height: 20 }, + { parent: g } + ); + e.selection.set([f, g]); + e.palette.run('Scale selection', '200'); + // The selection spans x 0..300, y 0..100: centre (150, 50). + assert.deepEqual(box(e, f), [-150, -50, 200, 200]); + assert.deepEqual(box(e, inner), [-130, -30, 40, 40]); + assert.deepEqual(box(e, g1), [250, -50, 40, 40]); + assert.deepEqual(box(e, g2), [410, 110, 40, 40]); + assert.deepEqual(box(e, g), [250, -50, 200, 200]); +}); + +test('it is one step to undo', () => { + const { e, a, b, c } = row(); + e.palette.run('Scale selection', '300'); + assert.ok(e.history.undo()); + assert.deepEqual( + [a, b, c].map((id) => box(e, id)), + [ + [0, 0, 50, 50], + [100, 0, 50, 50], + [200, 0, 50, 50], + ] + ); +}); + +test('anything but a positive number is refused', () => { + const { e, a } = row(); + for (const bad of ['abc', '0', '-50', '']) + e.palette.run('Scale selection', bad); + e.palette.run('Scale selection'); + assert.deepEqual(box(e, a), [0, 0, 50, 50]); + assert.equal(e.history.size, 0); +}); + +test('it is offered only with a selection', () => { + const e = createEditor(); + e.create('rect'); + assert.ok(!e.palette.list().includes('Scale selection')); +}); diff --git a/benchmarks/proof-build/hidden/B-V2/constraints.test.ts b/benchmarks/proof-build/hidden/B-V2/constraints.test.ts new file mode 100644 index 00000000..dcb2ef45 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V2/constraints.test.ts @@ -0,0 +1,47 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +const near = (actual: number, expected: number) => + assert.ok(Math.abs(actual - expected) < 1e-9, `${actual} is not ${expected}`); + +// Each test checks the rule only when the build has the command. + +test('scaled geometry is stored exactly; only the panel rounds it', () => { + const e = createEditor(); + const r = e.create('rect', { x: 0, y: 0, width: 10, height: 10 }); + e.selection.set([r]); + if (!e.palette.has('Scale selection')) return; + e.palette.run('Scale selection', '12.345'); + const n = e.doc.require(r); + near(n.width, 1.2345); + near(n.height, 1.2345); + near(n.x, 4.38275); + near(n.y, 4.38275); + assert.equal(e.panel.value('W'), '1.23'); +}); + +test('scaling up and back down returns the same numbers', () => { + const e = createEditor(); + const a = e.create('rect', { + x: 3.125, + y: 7.375, + width: 11.0625, + height: 13.4375, + }); + const b = e.create('rect', { x: 41, y: 2.5, width: 17, height: 5 }); + e.selection.set([a, b]); + if (!e.palette.has('Scale selection')) return; + e.palette.run('Scale selection', '150'); + e.palette.run('Scale selection', '100 / 1.5'); + for (const [id, x, y, w, h] of [ + [a, 3.125, 7.375, 11.0625, 13.4375], + [b, 41, 2.5, 17, 5], + ] as const) { + const n = e.doc.require(id); + near(n.x, x); + near(n.y, y); + near(n.width, w); + near(n.height, h); + } +}); diff --git a/benchmarks/proof-build/hidden/B-V2/naive.patch b/benchmarks/proof-build/hidden/B-V2/naive.patch new file mode 100644 index 00000000..a544a755 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V2/naive.patch @@ -0,0 +1,77 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..5ba1c70 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,5 +1,5 @@ + import { fail } from '../util/assert.ts'; +-import type { EditorContext } from './types.ts'; ++import type { EditorContext, Json } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; +@@ -9,6 +9,12 @@ import { lockItems } from './builtin/lock.ts'; + import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; ++import { descendants, topmostOnly } from '../model/tree.ts'; ++import { selectionBounds } from '../selection/bounds.ts'; ++import { centre } from '../model/geometry.ts'; ++import { parseExpression } from '../units/expr.ts'; ++import { roundTo } from '../util/math.ts'; ++import type { GeometryItem } from './builtin/resize.ts'; + import { isInteractive } from '../selection/rules.ts'; + + /** A command as the palette lists it, by its title. */ +@@ -57,6 +63,40 @@ export class Palette { + + const hasSelection = (ctx: EditorContext) => !ctx.selection.isEmpty(); + ++/** ++ * Scales the selected layers, and everything inside them, by `percent` ++ * about the centre of the selection, as one resize command. Results are ++ * rounded to hundredths so scaled layers land on tidy numbers. ++ */ ++function scaleSelection(ctx: EditorContext, input?: string): void { ++ const percent = parseExpression((input ?? '').replace(/%\s*$/, '')); ++ if (percent === null || !(percent > 0)) return; ++ const { doc } = ctx; ++ const ids = topmostOnly( ++ doc, ++ ctx.selection.ids().filter((id) => doc.has(id)) ++ ); ++ const bounds = selectionBounds(doc, ids); ++ if (!bounds) return; ++ const c = centre(bounds); ++ const k = percent / 100; ++ const items: GeometryItem[] = []; ++ for (const id of ids) ++ for (const node of [doc.require(id), ...descendants(doc, id)]) ++ if (node.type !== 'group') ++ items.push({ ++ id: node.id, ++ x: roundTo(c.x + (node.x - c.x) * k, 2), ++ y: roundTo(c.y + (node.y - c.y) * k, 2), ++ width: roundTo(node.width * k, 2), ++ height: roundTo(node.height * k, 2), ++ }); ++ if (items.length) ++ ctx.dispatch(command('resize', { items: items as unknown as Json[] }), { ++ label: 'Scale selection', ++ }); ++} ++ + function align(title: string, how: Alignment): PaletteEntry { + return { + title, +@@ -142,6 +182,11 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + align('Align bottom', 'bottom'), + distribute('Distribute horizontally', 'horizontal'), + distribute('Distribute vertically', 'vertical'), ++ { ++ title: 'Scale selection', ++ enabled: hasSelection, ++ run: (ctx, input) => scaleSelection(ctx, input), ++ }, + { + title: 'Hide', + enabled: hasSelection, diff --git a/benchmarks/proof-build/hidden/B-V2/reference.patch b/benchmarks/proof-build/hidden/B-V2/reference.patch new file mode 100644 index 00000000..ddfdefe6 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V2/reference.patch @@ -0,0 +1,76 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..2768772 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,5 +1,5 @@ + import { fail } from '../util/assert.ts'; +-import type { EditorContext } from './types.ts'; ++import type { EditorContext, Json } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; +@@ -9,6 +9,11 @@ import { lockItems } from './builtin/lock.ts'; + import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; ++import { descendants, topmostOnly } from '../model/tree.ts'; ++import { selectionBounds } from '../selection/bounds.ts'; ++import { centre } from '../model/geometry.ts'; ++import { parseExpression } from '../units/expr.ts'; ++import type { GeometryItem } from './builtin/resize.ts'; + import { isInteractive } from '../selection/rules.ts'; + + /** A command as the palette lists it, by its title. */ +@@ -57,6 +62,40 @@ export class Palette { + + const hasSelection = (ctx: EditorContext) => !ctx.selection.isEmpty(); + ++/** ++ * Scales the selected layers, and everything inside them, by `percent` ++ * about the centre of the selection, as one resize command. Stored values ++ * are not rounded. ++ */ ++function scaleSelection(ctx: EditorContext, input?: string): void { ++ const percent = parseExpression((input ?? '').replace(/%\s*$/, '')); ++ if (percent === null || !(percent > 0)) return; ++ const { doc } = ctx; ++ const ids = topmostOnly( ++ doc, ++ ctx.selection.ids().filter((id) => doc.has(id)) ++ ); ++ const bounds = selectionBounds(doc, ids); ++ if (!bounds) return; ++ const c = centre(bounds); ++ const k = percent / 100; ++ const items: GeometryItem[] = []; ++ for (const id of ids) ++ for (const node of [doc.require(id), ...descendants(doc, id)]) ++ if (node.type !== 'group') ++ items.push({ ++ id: node.id, ++ x: c.x + (node.x - c.x) * k, ++ y: c.y + (node.y - c.y) * k, ++ width: node.width * k, ++ height: node.height * k, ++ }); ++ if (items.length) ++ ctx.dispatch(command('resize', { items: items as unknown as Json[] }), { ++ label: 'Scale selection', ++ }); ++} ++ + function align(title: string, how: Alignment): PaletteEntry { + return { + title, +@@ -142,6 +181,11 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + align('Align bottom', 'bottom'), + distribute('Distribute horizontally', 'horizontal'), + distribute('Distribute vertically', 'vertical'), ++ { ++ title: 'Scale selection', ++ enabled: hasSelection, ++ run: (ctx, input) => scaleSelection(ctx, input), ++ }, + { + title: 'Hide', + enabled: hasSelection, diff --git a/benchmarks/proof-build/hidden/B-V3/acceptance.test.ts b/benchmarks/proof-build/hidden/B-V3/acceptance.test.ts new file mode 100644 index 00000000..56eb7c7a --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V3/acceptance.test.ts @@ -0,0 +1,105 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const sel = (e: Editor) => [...e.selection.ids()].sort(); +const sorted = (ids: string[]) => [...ids].sort(); +const pos = (e: Editor, id: string) => [ + e.doc.require(id).x, + e.doc.require(id).y, +]; + +function grouped() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0, y: 0, width: 50, height: 50 }); + const b = e.create('rect', { + name: 'B', + x: 100, + y: 0, + width: 50, + height: 50, + }); + const c = e.create('rect', { + name: 'C', + x: 200, + y: 0, + width: 50, + height: 50, + }); + e.selection.set([a, c]); + e.palette.run('Group selection'); + const [g] = e.selection.ids(); + e.history.clear(); + return { e, a, b, c, g }; +} + +test('Ungroup puts the children in the group’s place and selects them', () => { + const { e, a, b, c, g } = grouped(); + assert.deepEqual([...e.doc.children(null)], [b, g]); + e.selection.set([g]); + assert.ok(e.palette.run('Ungroup')); + assert.ok(!e.doc.has(g)); + assert.deepEqual([...e.doc.children(null)], [b, a, c]); + assert.deepEqual(sel(e), sorted([a, c])); + assert.deepEqual(pos(e, a), [0, 0]); + assert.deepEqual(pos(e, c), [200, 0]); +}); + +test('Mod+Shift+G ungroups on the canvas', () => { + const { e, a, c, g } = grouped(); + e.selection.set([g]); + assert.ok(e.keys.press('Mod+Shift+G')); + assert.ok(!e.doc.has(g)); + assert.deepEqual(sel(e), sorted([a, c])); +}); + +test('inside a frame the children take the group’s place among its siblings', () => { + const e = createEditor(); + const f = e.create('frame', { width: 500, height: 500 }); + const x = e.create('rect', { name: 'X' }, { parent: f }); + const g = e.create('group', { name: 'G' }, { parent: f }); + const y = e.create('rect', { name: 'Y', x: 10 }, { parent: g }); + const z = e.create('ellipse', { name: 'Z', x: 20 }, { parent: g }); + const w = e.create('rect', { name: 'W' }, { parent: f }); + e.selection.set([g]); + e.palette.run('Ungroup'); + assert.deepEqual([...e.doc.children(f)], [x, y, z, w]); + assert.equal(e.doc.require(y).parent, f); +}); + +test('other selected layers stay selected, and every selected group is ungrouped', () => { + const e = createEditor(); + const g1 = e.create('group'); + const p = e.create('rect', { x: 0 }, { parent: g1 }); + const g2 = e.create('group'); + const q = e.create('rect', { x: 100 }, { parent: g2 }); + const r = e.create('rect', { x: 200 }); + e.selection.set([g1, r, g2]); + e.palette.run('Ungroup'); + assert.ok(!e.doc.has(g1) && !e.doc.has(g2)); + assert.deepEqual([...e.doc.children(null)], [p, q, r]); + assert.deepEqual(sel(e), sorted([p, q, r])); +}); + +test('one Undo puts the groups back as they were', () => { + const { e, a, b, c, g } = grouped(); + e.selection.set([g]); + e.palette.run('Ungroup'); + assert.ok(e.history.undo()); + assert.ok(e.doc.has(g)); + assert.equal(e.doc.require(g).type, 'group'); + assert.deepEqual([...e.doc.children(null)], [b, g]); + assert.deepEqual([...e.doc.children(g)], [a, c]); + assert.ok(e.history.redo()); + assert.deepEqual([...e.doc.children(null)], [b, a, c]); +}); + +test('Ungroup is offered only when a group is selected', () => { + const { e, a, b } = grouped(); + e.selection.set([b]); + assert.ok(!e.palette.list().includes('Ungroup')); + e.selection.clear(); + assert.equal(e.palette.run('Ungroup'), false); + assert.ok(e.doc.has(a)); +}); diff --git a/benchmarks/proof-build/hidden/B-V3/constraints.test.ts b/benchmarks/proof-build/hidden/B-V3/constraints.test.ts new file mode 100644 index 00000000..4adc4f76 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V3/constraints.test.ts @@ -0,0 +1,21 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +test('undo and redo of Ungroup restore the exact selection on each side', () => { + const e = createEditor(); + const b = e.create('rect', { x: 100 }); + const g = e.create('group'); + e.create('rect', { x: 0 }, { parent: g }); + e.create('rect', { x: 200 }, { parent: g }); + const other = e.create('ellipse', { x: 300 }); + if (!e.palette.has('Ungroup')) return; + e.selection.set([other, g, b]); + const before = e.selection.ids(); + e.palette.run('Ungroup'); + const after = e.selection.ids(); + assert.ok(e.history.undo()); + assert.deepEqual(e.selection.ids(), before); + assert.ok(e.history.redo()); + assert.deepEqual(e.selection.ids(), after); +}); diff --git a/benchmarks/proof-build/hidden/B-V3/naive.patch b/benchmarks/proof-build/hidden/B-V3/naive.patch new file mode 100644 index 00000000..291030f5 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V3/naive.patch @@ -0,0 +1,79 @@ +diff --git a/src/commands/builtin/ungroup.ts b/src/commands/builtin/ungroup.ts +new file mode 100644 +index 0000000..850590d +--- /dev/null ++++ b/src/commands/builtin/ungroup.ts +@@ -0,0 +1,35 @@ ++import { indexInParent } from '../../model/tree.ts'; ++import type { EditorContext } from '../types.ts'; ++import { command } from '../types.ts'; ++ ++/** ++ * Moves the children of every selected group into the group's parent, in ++ * the group's place, and deletes the emptied groups, as one history entry. ++ * The freed children are selected first, so they stay selected. Returns ++ * false when no group is selected. ++ */ ++export function ungroupSelection(ctx: EditorContext): boolean { ++ const { doc } = ctx; ++ const selected = ctx.selection.ids().filter((id) => doc.has(id)); ++ const groups = selected.filter((id) => doc.require(id).type === 'group'); ++ if (!groups.length) return false; ++ const after = selected.flatMap((id) => ++ groups.includes(id) ? [...doc.children(id)] : [id] ++ ); ++ ctx.selection.set(after); ++ ctx.batch('Ungroup', () => { ++ groups.forEach((group) => { ++ const parent = doc.require(group).parent; ++ for (const child of [...doc.children(group)]) ++ ctx.dispatch( ++ command('reorder', { ++ id: child, ++ parent, ++ index: indexInParent(doc, group), ++ }) ++ ); ++ ctx.dispatch(command('delete', { ids: [group] })); ++ }); ++ }); ++ return true; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..1b5495e 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -2,6 +2,7 @@ import { fail } from '../util/assert.ts'; + import type { EditorContext } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; ++import { ungroupSelection } from './builtin/ungroup.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; + import type { Arrange } from './builtin/reorder.ts'; + import { visibilityItems } from './builtin/visibility.ts'; +@@ -130,6 +131,13 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + enabled: hasSelection, + run: (ctx) => void groupNodes(ctx, ctx.selection.ids()), + }, ++ { ++ title: 'Ungroup', ++ hint: 'Mod+Shift+G', ++ enabled: (ctx) => ++ ctx.selection.ids().some((id) => ctx.doc.get(id)?.type === 'group'), ++ run: (ctx) => void ungroupSelection(ctx), ++ }, + arrange('Bring forward', 'forward', 'Mod+]'), + arrange('Send backward', 'backward', 'Mod+['), + arrange('Bring to front', 'front', 'Mod+Shift+]'), +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..e368343 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -25,6 +25,7 @@ export const canvasBindings: readonly Binding[] = [ + palette('Escape', 'Deselect all'), + palette('Mod+A', 'Select all'), + palette('Mod+G', 'Group selection'), ++ palette('Mod+Shift+G', 'Ungroup'), + palette('Mod+]', 'Bring forward'), + palette('Mod+[', 'Send backward'), + palette('Mod+Shift+]', 'Bring to front'), diff --git a/benchmarks/proof-build/hidden/B-V3/reference.patch b/benchmarks/proof-build/hidden/B-V3/reference.patch new file mode 100644 index 00000000..dbbbb65d --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V3/reference.patch @@ -0,0 +1,80 @@ +diff --git a/src/commands/builtin/ungroup.ts b/src/commands/builtin/ungroup.ts +new file mode 100644 +index 0000000..416bdd5 +--- /dev/null ++++ b/src/commands/builtin/ungroup.ts +@@ -0,0 +1,36 @@ ++import { indexInParent } from '../../model/tree.ts'; ++import type { EditorContext } from '../types.ts'; ++import { command } from '../types.ts'; ++ ++/** ++ * Moves the children of every selected group into the group's parent, in ++ * the group's place, and deletes the emptied groups, as one history entry. ++ * The freed children take each group's place in the selection. Returns ++ * false when no group is selected. ++ */ ++export function ungroupSelection(ctx: EditorContext): boolean { ++ const { doc } = ctx; ++ const selected = ctx.selection.ids().filter((id) => doc.has(id)); ++ const groups = selected.filter((id) => doc.require(id).type === 'group'); ++ if (!groups.length) return false; ++ const after = selected.flatMap((id) => ++ groups.includes(id) ? [...doc.children(id)] : [id] ++ ); ++ ctx.batch('Ungroup', () => { ++ groups.forEach((group, n) => { ++ const parent = doc.require(group).parent; ++ for (const child of [...doc.children(group)]) ++ ctx.dispatch( ++ command('reorder', { ++ id: child, ++ parent, ++ index: indexInParent(doc, group), ++ }) ++ ); ++ ctx.dispatch(command('delete', { ids: [group] }), { ++ select: n === groups.length - 1 ? after : undefined, ++ }); ++ }); ++ }); ++ return true; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..1b5495e 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -2,6 +2,7 @@ import { fail } from '../util/assert.ts'; + import type { EditorContext } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; ++import { ungroupSelection } from './builtin/ungroup.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; + import type { Arrange } from './builtin/reorder.ts'; + import { visibilityItems } from './builtin/visibility.ts'; +@@ -130,6 +131,13 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + enabled: hasSelection, + run: (ctx) => void groupNodes(ctx, ctx.selection.ids()), + }, ++ { ++ title: 'Ungroup', ++ hint: 'Mod+Shift+G', ++ enabled: (ctx) => ++ ctx.selection.ids().some((id) => ctx.doc.get(id)?.type === 'group'), ++ run: (ctx) => void ungroupSelection(ctx), ++ }, + arrange('Bring forward', 'forward', 'Mod+]'), + arrange('Send backward', 'backward', 'Mod+['), + arrange('Bring to front', 'front', 'Mod+Shift+]'), +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..e368343 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -25,6 +25,7 @@ export const canvasBindings: readonly Binding[] = [ + palette('Escape', 'Deselect all'), + palette('Mod+A', 'Select all'), + palette('Mod+G', 'Group selection'), ++ palette('Mod+Shift+G', 'Ungroup'), + palette('Mod+]', 'Bring forward'), + palette('Mod+[', 'Send backward'), + palette('Mod+Shift+]', 'Bring to front'), diff --git a/benchmarks/proof-build/hidden/B-V4/acceptance.test.ts b/benchmarks/proof-build/hidden/B-V4/acceptance.test.ts new file mode 100644 index 00000000..66a45e8c --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V4/acceptance.test.ts @@ -0,0 +1,117 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const top = (e: Editor) => [...e.doc.children(null)]; +const at = (e: Editor, id: string) => { + const n = e.doc.require(id); + return [n.x, n.y, n.width, n.height]; +}; + +function setup() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0, y: 0, width: 50, height: 50 }); + const b = e.create('rect', { + name: 'B', + x: 100, + y: 0, + width: 50, + height: 50, + }); + const c = e.create('rect', { + name: 'C', + x: 200, + y: 0, + width: 50, + height: 50, + }); + return { e, a, b, c }; +} + +test('Duplicate copies a layer 10 px right and down, in front of it, and selects the copy', () => { + const { e, a, b, c } = setup(); + e.selection.set([a]); + assert.ok(e.palette.run('Duplicate')); + const [copy] = e.selection.ids(); + assert.ok(copy && copy !== a); + assert.equal(e.doc.size, 4); + assert.deepEqual(at(e, copy), [10, 10, 50, 50]); + assert.deepEqual(at(e, a), [0, 0, 50, 50]); + assert.equal(e.doc.require(copy).name, 'A'); + assert.equal(e.doc.require(copy).type, 'rect'); + assert.deepEqual(top(e), [a, copy, b, c]); +}); + +test('Mod+D duplicates, and again duplicates the copies', () => { + const { e, a, b } = setup(); + e.selection.set([a, b]); + assert.ok(e.keys.press('Mod+D')); + const first = e.selection.ids(); + assert.equal(first.length, 2); + assert.ok(!first.includes(a) && !first.includes(b)); + e.keys.press('Mod+D'); + const second = e.selection.ids(); + assert.equal(e.doc.size, 7); + const xs = second.map((id) => e.doc.require(id).x).sort((p, q) => p - q); + assert.deepEqual(xs, [20, 120]); + for (const id of second) assert.equal(e.doc.require(id).y, 20); +}); + +test('a copied frame brings copies of everything inside it', () => { + const e = createEditor(); + const f = e.create('frame', { + name: 'Card', + x: 0, + y: 0, + width: 200, + height: 100, + }); + const t = e.create( + 'text', + { name: 'Label', x: 10, y: 10, text: 'Hi' }, + { parent: f } + ); + const g = e.create('group', { name: 'Icons' }, { parent: f }); + e.create( + 'ellipse', + { name: 'Dot', x: 50, y: 50, width: 10, height: 10 }, + { parent: g } + ); + e.selection.set([f]); + e.palette.run('Duplicate'); + const [copy] = e.selection.ids(); + assert.equal(e.doc.size, 8); + const kids = e.doc.children(copy).map((id) => e.doc.require(id)); + assert.deepEqual( + kids.map((k) => [k.type, k.name, k.x, k.y]), + [ + ['text', 'Label', 20, 20], + ['group', 'Icons', 60, 60], + ] + ); + assert.ok(!kids.some((k) => k.id === t || k.id === g)); + const dot = e.doc.require(e.doc.children(kids[1].id)[0]); + assert.deepEqual([dot.name, dot.x, dot.y], ['Dot', 60, 60]); + assert.equal(e.doc.require(t).x, 10); +}); + +test('Undo removes the copies and reselects the originals; Redo brings them back', () => { + const { e, a, b } = setup(); + e.selection.set([a, b]); + e.palette.run('Duplicate'); + assert.ok(e.history.undo()); + assert.equal(e.doc.size, 3); + assert.deepEqual(e.selection.ids(), [a, b]); + assert.ok(e.history.redo()); + assert.equal(e.doc.size, 5); + const copies = e.doc.all().filter((n) => n.x === 10 || n.x === 110); + assert.equal(copies.length, 2); +}); + +test('Duplicate is offered only with a selection', () => { + const { e } = setup(); + assert.ok(!e.palette.list().includes('Duplicate')); + assert.equal(e.palette.run('Duplicate'), false); + assert.equal(e.doc.size, 3); +}); diff --git a/benchmarks/proof-build/hidden/B-V4/constraints.test.ts b/benchmarks/proof-build/hidden/B-V4/constraints.test.ts new file mode 100644 index 00000000..2f7f4f0a --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V4/constraints.test.ts @@ -0,0 +1,83 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; +import type { Command } from '../../src/commands/types.ts'; + +// Each test checks the rule only when the build has Duplicate. + +/** + * An editor whose id counter is ahead of what its document needs: a layer + * was made and dropped, so a copy loaded from its file counts differently. + */ +function setup() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', x: 0, y: 0, width: 50, height: 50 }); + const f = e.create('frame', { + name: 'F', + x: 100, + y: 0, + width: 200, + height: 200, + }); + e.create('ellipse', { name: 'E', x: 120, y: 20 }, { parent: f }); + const gone = e.create('rect', { name: 'Gone' }); + e.doc.remove(gone); + return { e, a, f }; +} + +function record(e: ReturnType) { + const log: Command[] = []; + const atRun: string[] = []; + e.commands.on('executed', ({ command }) => { + log.push(command); + atRun.push(JSON.stringify(command)); + }); + return { log, atRun }; +} + +test('the recorded commands replay as JSON on a copy of the document', () => { + const { e, a, f } = setup(); + const saved = e.files.save(); + const { log } = record(e); + e.selection.set([a, f]); + if (!e.palette.has('Duplicate')) return; + e.palette.run('Duplicate'); + e.keys.press('Mod+D'); + const copy = createEditor(); + copy.files.load(saved); + copy.commands.replay(JSON.parse(JSON.stringify(log))); + assert.equal(copy.files.save(), e.files.save()); +}); + +test('a command keeps the payload it ran with', () => { + const { e, a } = setup(); + const { log, atRun } = record(e); + e.selection.set([a]); + if (!e.palette.has('Duplicate')) return; + e.palette.run('Duplicate'); + e.palette.run('Duplicate'); + e.commands.dispatch({ + type: 'move', + payload: { ids: e.selection.ids(), dx: 30, dy: 0 }, + }); + e.history.undo(); + e.history.undo(); + e.history.redo(); + assert.deepEqual( + log.map((c) => JSON.stringify(c)), + atRun + ); +}); + +test('Redo brings back the very same copies, selected', () => { + const { e, a, f } = setup(); + e.selection.set([a, f]); + if (!e.palette.has('Duplicate')) return; + e.palette.run('Duplicate'); + const copies = e.selection.ids(); + const saved = e.files.save(); + e.history.undo(); + e.history.redo(); + assert.deepEqual(e.selection.ids(), copies); + assert.equal(e.files.save(), saved); +}); diff --git a/benchmarks/proof-build/hidden/B-V4/naive.patch b/benchmarks/proof-build/hidden/B-V4/naive.patch new file mode 100644 index 00000000..090dc12d --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V4/naive.patch @@ -0,0 +1,120 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..7ba46c0 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,5 +1,5 @@ + import { fail } from '../util/assert.ts'; +-import type { EditorContext } from './types.ts'; ++import type { EditorContext, Json } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; +@@ -9,6 +9,14 @@ import { lockItems } from './builtin/lock.ts'; + import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; ++import { ++ descendants, ++ indexInParent, ++ sortByPaintOrder, ++ topmostOnly, ++} from '../model/tree.ts'; ++import type { SceneNode } from '../model/types.ts'; ++import { isContainer } from '../model/types.ts'; + import { isInteractive } from '../selection/rules.ts'; + + /** A command as the palette lists it, by its title. */ +@@ -57,6 +65,68 @@ export class Palette { + + const hasSelection = (ctx: EditorContext) => !ctx.selection.isEmpty(); + ++/** How far Duplicate moves each copy from its original. */ ++export const DUPLICATE_OFFSET = 10; ++ ++/** `id` and everything inside it, root first, with new ids, moved by (dx, dy). */ ++function copySubtree( ++ ctx: EditorContext, ++ id: string, ++ dx: number, ++ dy: number ++): SceneNode[] { ++ const nodes = [ctx.doc.require(id), ...descendants(ctx.doc, id)]; ++ const fresh = new Map(nodes.map((n) => [n.id, ctx.ids.next(n.type)])); ++ return nodes.map((n) => { ++ const copy = structuredClone(n); ++ copy.id = fresh.get(n.id)!; ++ copy.parent = ++ n.parent !== null && fresh.has(n.parent) ++ ? fresh.get(n.parent)! ++ : n.parent; ++ copy.x += dx; ++ copy.y += dy; ++ if (isContainer(copy)) ++ copy.children = copy.children.map((c) => fresh.get(c)!); ++ return copy; ++ }); ++} ++ ++/** ++ * Copies every selected layer (and what is inside it) in front of its ++ * original, offset by DUPLICATE_OFFSET, and selects the copies. Each copy ++ * is an insert command that carries the copied nodes with their new ids. ++ */ ++function duplicate(ctx: EditorContext): void { ++ const { doc } = ctx; ++ const originals = sortByPaintOrder( ++ doc, ++ topmostOnly( ++ doc, ++ ctx.selection.ids().filter((id) => doc.has(id)) ++ ) ++ ); ++ if (!originals.length) return; ++ const copies = originals.map((id) => ({ ++ id, ++ nodes: copySubtree(ctx, id, DUPLICATE_OFFSET, DUPLICATE_OFFSET), ++ })); ++ const select = copies.map((c) => c.nodes[0].id); ++ ctx.batch('Duplicate', () => { ++ // Front to back, so the indexes of the originals still to copy hold. ++ [...copies].reverse().forEach(({ id, nodes }) => { ++ ctx.dispatch( ++ command('insert', { ++ nodes: nodes as unknown as Json[], ++ parent: doc.require(id).parent, ++ index: indexInParent(doc, id) + 1, ++ }) ++ ); ++ }); ++ }); ++ ctx.selection.set(select); ++} ++ + function align(title: string, how: Alignment): PaletteEntry { + return { + title, +@@ -124,6 +194,12 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + }); + }, + }, ++ { ++ title: 'Duplicate', ++ hint: 'Mod+D', ++ enabled: hasSelection, ++ run: (ctx) => duplicate(ctx), ++ }, + { + title: 'Group selection', + hint: 'Mod+G', +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..c6e8fa3 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -25,6 +25,7 @@ export const canvasBindings: readonly Binding[] = [ + palette('Escape', 'Deselect all'), + palette('Mod+A', 'Select all'), + palette('Mod+G', 'Group selection'), ++ palette('Mod+D', 'Duplicate'), + palette('Mod+]', 'Bring forward'), + palette('Mod+[', 'Send backward'), + palette('Mod+Shift+]', 'Bring to front'), diff --git a/benchmarks/proof-build/hidden/B-V4/reference.patch b/benchmarks/proof-build/hidden/B-V4/reference.patch new file mode 100644 index 00000000..91671397 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V4/reference.patch @@ -0,0 +1,120 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..3924297 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,5 +1,5 @@ + import { fail } from '../util/assert.ts'; +-import type { EditorContext } from './types.ts'; ++import type { EditorContext, Json } from './types.ts'; + import { command } from './types.ts'; + import { groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; +@@ -9,6 +9,14 @@ import { lockItems } from './builtin/lock.ts'; + import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; ++import { ++ descendants, ++ indexInParent, ++ sortByPaintOrder, ++ topmostOnly, ++} from '../model/tree.ts'; ++import type { SceneNode } from '../model/types.ts'; ++import { isContainer } from '../model/types.ts'; + import { isInteractive } from '../selection/rules.ts'; + + /** A command as the palette lists it, by its title. */ +@@ -57,6 +65,68 @@ export class Palette { + + const hasSelection = (ctx: EditorContext) => !ctx.selection.isEmpty(); + ++/** How far Duplicate moves each copy from its original. */ ++export const DUPLICATE_OFFSET = 10; ++ ++/** `id` and everything inside it, root first, with new ids, moved by (dx, dy). */ ++function copySubtree( ++ ctx: EditorContext, ++ id: string, ++ dx: number, ++ dy: number ++): SceneNode[] { ++ const nodes = [ctx.doc.require(id), ...descendants(ctx.doc, id)]; ++ const fresh = new Map(nodes.map((n) => [n.id, ctx.ids.next(n.type)])); ++ return nodes.map((n) => { ++ const copy = structuredClone(n); ++ copy.id = fresh.get(n.id)!; ++ copy.parent = ++ n.parent !== null && fresh.has(n.parent) ++ ? fresh.get(n.parent)! ++ : n.parent; ++ copy.x += dx; ++ copy.y += dy; ++ if (isContainer(copy)) ++ copy.children = copy.children.map((c) => fresh.get(c)!); ++ return copy; ++ }); ++} ++ ++/** ++ * Copies every selected layer (and what is inside it) in front of its ++ * original, offset by DUPLICATE_OFFSET, and selects the copies. Each copy ++ * is an insert command that carries the copied nodes with their new ids. ++ */ ++function duplicate(ctx: EditorContext): void { ++ const { doc } = ctx; ++ const originals = sortByPaintOrder( ++ doc, ++ topmostOnly( ++ doc, ++ ctx.selection.ids().filter((id) => doc.has(id)) ++ ) ++ ); ++ if (!originals.length) return; ++ const copies = originals.map((id) => ({ ++ id, ++ nodes: copySubtree(ctx, id, DUPLICATE_OFFSET, DUPLICATE_OFFSET), ++ })); ++ const select = copies.map((c) => c.nodes[0].id); ++ ctx.batch('Duplicate', () => { ++ // Front to back, so the indexes of the originals still to copy hold. ++ [...copies].reverse().forEach(({ id, nodes }, i) => { ++ ctx.dispatch( ++ command('insert', { ++ nodes: nodes as unknown as Json[], ++ parent: doc.require(id).parent, ++ index: indexInParent(doc, id) + 1, ++ }), ++ { select: i === copies.length - 1 ? select : undefined } ++ ); ++ }); ++ }); ++} ++ + function align(title: string, how: Alignment): PaletteEntry { + return { + title, +@@ -124,6 +194,12 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + }); + }, + }, ++ { ++ title: 'Duplicate', ++ hint: 'Mod+D', ++ enabled: hasSelection, ++ run: (ctx) => duplicate(ctx), ++ }, + { + title: 'Group selection', + hint: 'Mod+G', +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..c6e8fa3 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -25,6 +25,7 @@ export const canvasBindings: readonly Binding[] = [ + palette('Escape', 'Deselect all'), + palette('Mod+A', 'Select all'), + palette('Mod+G', 'Group selection'), ++ palette('Mod+D', 'Duplicate'), + palette('Mod+]', 'Bring forward'), + palette('Mod+[', 'Send backward'), + palette('Mod+Shift+]', 'Bring to front'), diff --git a/benchmarks/proof-build/hidden/B-V5/acceptance.test.ts b/benchmarks/proof-build/hidden/B-V5/acceptance.test.ts new file mode 100644 index 00000000..e9026c32 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V5/acceptance.test.ts @@ -0,0 +1,68 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + let now = 0; + const e = createEditor({ clock: () => (now += 16) }); + const a = e.create('rect', { x: 0, y: 0, width: 50, height: 50 }); + e.selection.set([a]); + return { e, a }; +} + +function hold( + e: ReturnType, + chord: string, + repeats: number +) { + e.keys.down(chord); + for (let i = 0; i < repeats; i++) e.keys.down(chord, { repeat: true }); + e.keys.up(chord); +} + +test('holding an arrow key keeps nudging', () => { + const { e, a } = setup(); + hold(e, 'ArrowRight', 5); + assert.equal(e.doc.require(a).x, 6); + hold(e, 'Shift+ArrowDown', 2); + assert.equal(e.doc.require(a).y, 30); +}); + +test('one Undo takes back the whole hold', () => { + const { e, a } = setup(); + const before = e.history.size; + hold(e, 'ArrowLeft', 20); + assert.equal(e.doc.require(a).x, -21); + assert.equal(e.history.size, before + 1); + assert.ok(e.history.undo()); + assert.equal(e.doc.require(a).x, 0); + assert.ok(e.history.redo()); + assert.equal(e.doc.require(a).x, -21); +}); + +test('a hold with Shift undoes in one step too', () => { + const { e, a } = setup(); + hold(e, 'Shift+ArrowUp', 4); + assert.equal(e.doc.require(a).y, -50); + e.history.undo(); + assert.equal(e.doc.require(a).y, 0); +}); + +test('separate presses are separate steps', () => { + const { e, a } = setup(); + e.keys.press('ArrowRight'); + e.keys.press('ArrowRight'); + e.keys.press('ArrowRight'); + assert.equal(e.doc.require(a).x, 3); + e.history.undo(); + assert.equal(e.doc.require(a).x, 2); +}); + +test('a hold after a press is its own step', () => { + const { e, a } = setup(); + e.keys.press('ArrowRight'); + hold(e, 'ArrowRight', 3); + assert.equal(e.doc.require(a).x, 5); + e.history.undo(); + assert.equal(e.doc.require(a).x, 1); +}); diff --git a/benchmarks/proof-build/hidden/B-V5/constraints.test.ts b/benchmarks/proof-build/hidden/B-V5/constraints.test.ts new file mode 100644 index 00000000..45b83b17 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V5/constraints.test.ts @@ -0,0 +1,54 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +/** An editor whose clock the test moves by hand. */ +function setup() { + const clock = { now: 0 }; + const e = createEditor({ clock: () => clock.now }); + const a = e.create('rect', { x: 0, y: 0, width: 50, height: 50 }); + e.selection.set([a]); + return { e, a, clock }; +} + +/** History entries one hold of `chord` makes, with `gap` ms between repeats. */ +function entriesForHold(chord: string, repeats: number, gap: number) { + const { e, clock } = setup(); + e.keys.down(chord); + for (let i = 0; i < repeats; i++) { + clock.now += gap; + e.keys.down(chord, { repeat: true }); + } + e.keys.up(chord); + return e.history.size; +} + +test('how fast a key repeats does not change what a hold becomes in history', () => { + for (const chord of ['ArrowRight', 'Shift+ArrowDown']) + assert.equal( + entriesForHold(chord, 4, 1500), + entriesForHold(chord, 4, 16), + chord + ); +}); + +test('two holds a moment apart make twice the entries of one', () => { + const one = entriesForHold('ArrowDown', 1, 1); + const { e, clock } = setup(); + for (let n = 0; n < 2; n++) { + e.keys.down('ArrowDown'); + clock.now += 1; + e.keys.down('ArrowDown', { repeat: true }); + e.keys.up('ArrowDown'); + clock.now += 1; + } + assert.equal(e.history.size, 2 * one); +}); + +test('separate presses with no time between them stay apart', () => { + const { e, a } = setup(); + for (let i = 0; i < 3; i++) e.keys.press('ArrowLeft'); + assert.equal(e.history.size, 3); + e.history.undo(); + assert.equal(e.doc.require(a).x, -2); +}); diff --git a/benchmarks/proof-build/hidden/B-V5/naive.patch b/benchmarks/proof-build/hidden/B-V5/naive.patch new file mode 100644 index 00000000..89051136 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V5/naive.patch @@ -0,0 +1,59 @@ +diff --git a/src/commands/types.ts b/src/commands/types.ts +index e3308ec..a84b366 100644 +--- a/src/commands/types.ts ++++ b/src/commands/types.ts +@@ -43,6 +43,8 @@ export interface EditorContext { + dispatch(command: Command, options?: DispatchOptions): void; + batch(label: string, run: () => void): void; + newGesture(): string; ++ /** The editor's clock, in ms. */ ++ now(): number; + undo(): boolean; + redo(): boolean; + } +diff --git a/src/editor.ts b/src/editor.ts +index 3ff92fa..7272a63 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -65,6 +65,7 @@ export function createEditor(options: EditorOptions = {}) { + dispatch: (c: Command, o?: DispatchOptions) => bus.dispatch(c, o), + batch: (label: string, run: () => void) => batch(bus, label, run), + newGesture: () => gestures.next(), ++ now: options.clock ?? Date.now, + undo: () => bus.undo(), + redo: () => bus.redo(), + }; +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..832513a 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -8,13 +8,27 @@ function palette(chord: string, title: string): Binding { + return { chord, run: (ctx) => void ctx.palette.run(title) }; + } + ++/** Key repeats closer together than this belong to one hold. */ ++export const REPEAT_WINDOW_MS = 400; ++ ++const lastNudge = new WeakMap(); ++ + function nudge(chord: string, direction: Direction, large: boolean): Binding { + return { + chord, +- run(ctx: KeyContext) { ++ run(ctx: KeyContext, event) { + const ids = ctx.selection.ids(); + if (!ids.length) return; +- ctx.dispatch(command('nudge', { ids, ...nudgeDelta(direction, large) })); ++ const now = ctx.now(); ++ const last = lastNudge.get(ctx); ++ const gesture = ++ event.repeat && last && now - last.at <= REPEAT_WINDOW_MS ++ ? last.gesture ++ : ctx.newGesture(); ++ lastNudge.set(ctx, { gesture, at: now }); ++ ctx.dispatch(command('nudge', { ids, ...nudgeDelta(direction, large) }), { ++ gesture, ++ }); + }, + }; + } diff --git a/benchmarks/proof-build/hidden/B-V5/reference.patch b/benchmarks/proof-build/hidden/B-V5/reference.patch new file mode 100644 index 00000000..9b218025 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-V5/reference.patch @@ -0,0 +1,36 @@ +diff --git a/src/keys/bindings/canvas.ts b/src/keys/bindings/canvas.ts +index d550730..4568aff 100644 +--- a/src/keys/bindings/canvas.ts ++++ b/src/keys/bindings/canvas.ts +@@ -8,13 +8,29 @@ function palette(chord: string, title: string): Binding { + return { chord, run: (ctx) => void ctx.palette.run(title) }; + } + ++/** The gesture of each arrow key being held, per editor. */ ++const holds = new WeakMap>(); ++ ++/** ++ * A nudge binding. A press starts a gesture and the key's repeats join it, ++ * so a whole hold is one history entry; releasing the key ends it. ++ */ + function nudge(chord: string, direction: Direction, large: boolean): Binding { + return { + chord, +- run(ctx: KeyContext) { ++ run(ctx: KeyContext, event) { + const ids = ctx.selection.ids(); + if (!ids.length) return; +- ctx.dispatch(command('nudge', { ids, ...nudgeDelta(direction, large) })); ++ let held = holds.get(ctx); ++ if (!held) holds.set(ctx, (held = new Map())); ++ let gesture = event.repeat ? held.get(event.chord) : undefined; ++ if (!gesture) held.set(event.chord, (gesture = ctx.newGesture())); ++ ctx.dispatch(command('nudge', { ids, ...nudgeDelta(direction, large) }), { ++ gesture, ++ }); ++ }, ++ release(ctx: KeyContext, event) { ++ holds.get(ctx)?.delete(event.chord); + }, + }; + } diff --git a/benchmarks/proof-build/hidden/_global/global-rules.test.ts b/benchmarks/proof-build/hidden/_global/global-rules.test.ts new file mode 100644 index 00000000..0c948bfd --- /dev/null +++ b/benchmarks/proof-build/hidden/_global/global-rules.test.ts @@ -0,0 +1,645 @@ +// Invariants that hold rules only Proof states: a shipped migration step +// never changes its output (R3), and one palette command run is at most +// one history entry (R1). grade.mjs runs this file only with proof-only +// and stale tasks (see globals.json). The expected values were computed +// from the editor as it was before any task. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; +import { MIGRATIONS } from '../../src/persist/migrations/index.ts'; + +const SHIPPED_STEPS: { from: number; input: unknown; output: unknown }[] = [ + { + from: 1, + input: { + version: 1, + name: 'Pricing card', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: '0px', + y: '0px', + width: '320px', + height: '200px', + fill: '#ffffff', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: '16px', + y: '16', + width: '48px', + height: '24px', + fill: '#3366ff', + stroke: '#000000', + strokeWidth: '2px', + }, + { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: '16px', + y: '56px', + width: '200px', + height: '18px', + fontSize: '14px', + text: 'Most popular', + fill: '#111111', + hidden: true, + }, + ], + }, + output: { + version: 2, + name: 'Pricing card', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fill: '#ffffff', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fill: '#3366ff', + stroke: '#000000', + strokeWidth: 2, + }, + { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + fill: '#111111', + hidden: true, + }, + ], + }, + }, + { + from: 2, + input: { + version: 2, + name: 'Pricing card', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fill: '#ffffff', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fill: '#3366ff', + stroke: '#000000', + strokeWidth: 2, + }, + { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + fill: '#111111', + hidden: true, + }, + ], + }, + output: { + version: 3, + name: 'Pricing card', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fills: [{ color: '#FFFFFF' }], + strokes: [], + }, + { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fills: [{ color: '#3366FF' }], + strokes: [{ color: '#000000', width: 2, align: 'inside' }], + }, + { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + hidden: true, + fills: [{ color: '#111111' }], + strokes: [], + }, + ], + }, + }, + { + from: 3, + input: { + version: 3, + name: 'Pricing card', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fills: [{ color: '#FFFFFF' }], + strokes: [], + }, + { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fills: [{ color: '#3366FF' }], + strokes: [{ color: '#000000', width: 2, align: 'inside' }], + }, + { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + hidden: true, + fills: [{ color: '#111111' }], + strokes: [], + }, + ], + }, + output: { + version: 4, + name: 'Pricing card', + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fills: [{ color: '#FFFFFF' }], + strokes: [], + visible: true, + locked: false, + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fills: [{ color: '#3366FF' }], + strokes: [{ color: '#000000', width: 2, align: 'inside' }], + visible: true, + locked: false, + }, + 'text-3': { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + fills: [{ color: '#111111' }], + strokes: [], + visible: false, + locked: false, + }, + }, + top: ['frame-1'], + }, + }, + { + from: 4, + input: { + version: 4, + name: 'Pricing card', + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fills: [{ color: '#FFFFFF' }], + strokes: [], + visible: true, + locked: false, + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fills: [{ color: '#3366FF' }], + strokes: [{ color: '#000000', width: 2, align: 'inside' }], + visible: true, + locked: false, + }, + 'text-3': { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + fills: [{ color: '#111111' }], + strokes: [], + visible: false, + locked: false, + }, + }, + top: ['frame-1'], + }, + output: { + version: 5, + name: 'Pricing card', + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Card', + parent: null, + children: ['rect-2', 'text-3'], + x: 0, + y: 0, + width: 320, + height: 200, + fills: [{ color: '#FFFFFF' }], + strokes: [], + visible: true, + locked: false, + rotation: 0, + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Badge', + parent: 'frame-1', + x: 16, + y: 16, + width: 48, + height: 24, + fills: [{ color: '#3366FF' }], + strokes: [{ color: '#000000', width: 2, align: 'inside' }], + visible: true, + locked: false, + rotation: 0, + }, + 'text-3': { + id: 'text-3', + type: 'text', + name: 'Label', + parent: 'frame-1', + x: 16, + y: 56, + width: 200, + height: 18, + fontSize: 14, + text: 'Most popular', + fills: [{ color: '#111111' }], + strokes: [], + visible: false, + locked: false, + rotation: 0, + }, + }, + top: ['frame-1'], + }, + }, + { + from: 4, + input: { + version: 4, + name: 'Rotated', + top: ['frame-1'], + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Board', + parent: null, + children: ['rect-2'], + x: 0, + y: 0, + width: 400, + height: 300, + rotation: 0, + visible: true, + locked: false, + opacity: 1, + clip: true, + fills: [{ color: '#FFFFFF' }], + strokes: [], + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Tilted', + parent: 'frame-1', + x: 100, + y: 100, + width: 80, + height: 40, + rotation: 45, + visible: true, + locked: false, + opacity: 1, + fills: [{ color: '#FF0000' }], + strokes: [], + }, + }, + }, + output: { + version: 5, + name: 'Rotated', + top: ['frame-1'], + nodes: { + 'frame-1': { + id: 'frame-1', + type: 'frame', + name: 'Board', + parent: null, + children: ['rect-2'], + x: 0, + y: 0, + width: 400, + height: 300, + rotation: 0, + visible: true, + locked: false, + opacity: 1, + clip: true, + fills: [{ color: '#FFFFFF' }], + strokes: [], + }, + 'rect-2': { + id: 'rect-2', + type: 'rect', + name: 'Tilted', + parent: 'frame-1', + x: 100, + y: 100, + width: 80, + height: 40, + rotation: 0.7853981633974483, + visible: true, + locked: false, + opacity: 1, + fills: [{ color: '#FF0000' }], + strokes: [], + }, + }, + }, + }, + { + from: 1, + input: { + version: 1, + name: 'Legacy units', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Sheet', + parent: null, + children: ['rect-2'], + x: '0px', + y: '0', + width: '8.5in', + height: '11in', + fill: '#FFFFFF', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Margin', + parent: 'frame-1', + x: '50%', + y: '18pt', + width: '2in', + height: ' 12 px ', + strokeWidth: '1pt', + stroke: '#000000', + }, + ], + }, + output: { + version: 2, + name: 'Legacy units', + nodes: [ + { + id: 'frame-1', + type: 'frame', + name: 'Sheet', + parent: null, + children: ['rect-2'], + x: 0, + y: 0, + width: 0, + height: 0, + fill: '#FFFFFF', + }, + { + id: 'rect-2', + type: 'rect', + name: 'Margin', + parent: 'frame-1', + x: 0, + y: 0, + width: 0, + height: 12, + strokeWidth: 0, + stroke: '#000000', + }, + ], + }, + }, +]; + +test('shipped migration steps return what they always returned', () => { + for (const { from, input, output } of SHIPPED_STEPS) { + const step = MIGRATIONS.find((m) => m.from === from && m.to === from + 1); + assert.ok(step, `a step from version ${from}`); + assert.deepEqual( + step.run(structuredClone(input) as Record), + output, + `step ${from}→${from + 1}` + ); + } +}); + +function sampleEditor() { + const e = createEditor(); + const frame = e.create('frame', { + name: 'Card', + x: 0, + y: 0, + width: 400, + height: 300, + }); + e.create( + 'rect', + { name: 'Back', x: 10, y: 10, width: 50, height: 40 }, + { parent: frame } + ); + const group = e.create('group', { name: 'Icons' }, { parent: frame }); + e.create( + 'ellipse', + { name: 'Dot', x: 100, y: 100, width: 20, height: 20 }, + { parent: group } + ); + e.create( + 'rect', + { name: 'Bar', x: 140, y: 100, width: 60, height: 20, rotation: 0.5 }, + { parent: group } + ); + e.create( + 'text', + { name: 'Title', text: 'Hello there', x: 20, y: 200 }, + { parent: frame } + ); + e.create('line', { name: 'Rule', x: 0, y: 320, width: 400 }); + const loose = e.create('rect', { + name: 'Loose', + x: 500, + y: 0, + width: 80, + height: 80, + opacity: 0.5, + }); + return { e, frame, group, loose }; +} + +test('every palette command makes at most one history entry per run', () => { + const setups: ((s: ReturnType) => string[])[] = [ + ({ e, frame }) => [...e.doc.children(frame)], + ({ group, loose }) => [group, loose], + ({ loose }) => [loose], + ({ e }) => [...e.doc.children(null)], + ]; + for (const pick of setups) + for (const input of [undefined, '150']) { + const titles = (() => { + const s = sampleEditor(); + s.e.selection.set(pick(s)); + return s.e.palette.list(); + })(); + for (const title of titles) { + const s = sampleEditor(); + s.e.selection.set(pick(s)); + const before = s.e.history.size; + try { + s.e.palette.run(title, input); + } catch { + continue; + } + assert.ok( + s.e.history.size - before <= 1, + `"${title}" (input ${input}) made ${ + s.e.history.size - before + } entries` + ); + } + } +}); diff --git a/benchmarks/proof-build/hidden/_global/global.test.ts b/benchmarks/proof-build/hidden/_global/global.test.ts new file mode 100644 index 00000000..61214e19 --- /dev/null +++ b/benchmarks/proof-build/hidden/_global/global.test.ts @@ -0,0 +1,125 @@ +// Invariants every task keeps, whatever it builds, and that the code itself +// states: saving is stable, and plugin API v1 is frozen. They run with each +// task's constraint tests. Written against the editor as it was before any +// task; the expected values below were computed from that tree. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +/** The version 5 fixture document as it shipped. */ +const DOC_V5 = + '{\n "version": 5,\n "name": "Landing page",\n "top": ["frame-1", "rect-9"],\n "nodes": {\n "frame-1": {\n "children": ["text-2", "text-3", "group-4", "ellipse-7", "line-8"],\n "clip": true,\n "fills": [\n {\n "color": "#F4F1EA"\n }\n ],\n "height": 640,\n "id": "frame-1",\n "locked": false,\n "name": "Hero",\n "opacity": 1,\n "parent": null,\n "rotation": 0,\n "strokes": [],\n "type": "frame",\n "visible": true,\n "width": 1200,\n "x": 0,\n "y": 0\n },\n "text-2": {\n "align": "left",\n "fills": [\n {\n "color": "#000000"\n }\n ],\n "fontSize": 48,\n "height": 60,\n "id": "text-2",\n "lineHeight": 1.2,\n "locked": false,\n "name": "Headline",\n "opacity": 1,\n "parent": "frame-1",\n "rotation": 0,\n "strokes": [],\n "text": "Make something people want",\n "type": "text",\n "visible": true,\n "width": 640,\n "x": 80,\n "y": 120\n },\n "text-3": {\n "align": "left",\n "fills": [\n {\n "color": "#000000"\n }\n ],\n "fontSize": 20,\n "height": 48,\n "id": "text-3",\n "lineHeight": 1.25,\n "locked": false,\n "name": "Subhead",\n "opacity": 1,\n "parent": "frame-1",\n "rotation": 0,\n "strokes": [],\n "text": "A short line under the headline.",\n "type": "text",\n "visible": true,\n "width": 520,\n "x": 80,\n "y": 200\n },\n "group-4": {\n "children": ["rect-5", "rect-6"],\n "height": 48,\n "id": "group-4",\n "locked": false,\n "name": "Buttons",\n "opacity": 0.9,\n "parent": "frame-1",\n "rotation": 0,\n "type": "group",\n "visible": true,\n "width": 340,\n "x": 80,\n "y": 300\n },\n "rect-5": {\n "fills": [\n {\n "color": "#1F4FFF"\n }\n ],\n "height": 48,\n "id": "rect-5",\n "locked": false,\n "name": "Primary",\n "opacity": 1,\n "parent": "group-4",\n "rotation": 0,\n "strokes": [],\n "type": "rect",\n "visible": true,\n "width": 160,\n "x": 80,\n "y": 300\n },\n "rect-6": {\n "fills": [\n {\n "color": "#FFFFFF"\n }\n ],\n "height": 48,\n "id": "rect-6",\n "locked": false,\n "name": "Secondary",\n "opacity": 1,\n "parent": "group-4",\n "rotation": 0,\n "strokes": [\n {\n "color": "#1F4FFF",\n "width": 2,\n "align": "inside"\n }\n ],\n "type": "rect",\n "visible": true,\n "width": 160,\n "x": 260,\n "y": 300\n },\n "ellipse-7": {\n "fills": [\n {\n "color": "#FFB340"\n }\n ],\n "height": 300,\n "id": "ellipse-7",\n "locked": false,\n "name": "Blob",\n "opacity": 0.6,\n "parent": "frame-1",\n "rotation": 0.5235987755982988,\n "strokes": [],\n "type": "ellipse",\n "visible": true,\n "width": 300,\n "x": 820,\n "y": 140\n },\n "line-8": {\n "fills": [],\n "height": 0,\n "id": "line-8",\n "locked": true,\n "name": "Divider",\n "opacity": 1,\n "parent": "frame-1",\n "rotation": 0,\n "strokes": [\n {\n "color": "#000000",\n "width": 1,\n "align": "center"\n }\n ],\n "type": "line",\n "visible": true,\n "width": 1040,\n "x": 80,\n "y": 400\n },\n "rect-9": {\n "fills": [\n {\n "color": "#D9D9D9"\n }\n ],\n "height": 80,\n "id": "rect-9",\n "locked": false,\n "name": "Hidden note",\n "opacity": 1,\n "parent": null,\n "rotation": 0,\n "strokes": [],\n "type": "rect",\n "visible": false,\n "width": 200,\n "x": 0,\n "y": 700\n }\n }\n}\n'; + +/** What api.shortcuts() returned in plugin API v1. */ +const V1_SHORTCUTS = { + global: { 'Mod+Z': 'undo', 'Mod+Shift+Z': 'redo' }, + canvas: { + Delete: 'delete', + Backspace: 'delete', + Escape: 'deselect-all', + 'Mod+A': 'select-all', + 'Mod+G': 'group', + 'Mod+]': 'bring-forward', + 'Mod+[': 'send-backward', + 'Mod+Shift+]': 'bring-to-front', + 'Mod+Shift+[': 'send-to-back', + 'Mod+Shift+L': 'toggle-lock', + ArrowLeft: 'nudge-left', + ArrowRight: 'nudge-right', + ArrowUp: 'nudge-up', + ArrowDown: 'nudge-down', + 'Shift+ArrowLeft': 'nudge-left-large', + 'Shift+ArrowRight': 'nudge-right-large', + 'Shift+ArrowUp': 'nudge-up-large', + 'Shift+ArrowDown': 'nudge-down-large', + }, + layers: { + ArrowUp: 'select-previous', + ArrowDown: 'select-next', + ArrowRight: 'expand', + ArrowLeft: 'collapse', + Delete: 'delete', + 'Mod+G': 'group', + }, + panel: { Escape: 'focus-canvas' }, +}; + +const V1_SURFACE = [ + 'export', + 'getSelection', + 'listCommands', + 'name', + 'off', + 'on', + 'runCommand', + 'shortcuts', + 'version', +]; + +function sampleEditor() { + const e = createEditor(); + const frame = e.create('frame', { + name: 'Card', + x: 0, + y: 0, + width: 400, + height: 300, + }); + e.create( + 'rect', + { name: 'Back', x: 10, y: 10, width: 50, height: 40 }, + { parent: frame } + ); + const group = e.create('group', { name: 'Icons' }, { parent: frame }); + e.create( + 'ellipse', + { name: 'Dot', x: 100, y: 100, width: 20, height: 20 }, + { parent: group } + ); + e.create( + 'rect', + { name: 'Bar', x: 140, y: 100, width: 60, height: 20, rotation: 0.5 }, + { parent: group } + ); + e.create( + 'text', + { name: 'Title', text: 'Hello there', x: 20, y: 200 }, + { parent: frame } + ); + e.create('line', { name: 'Rule', x: 0, y: 320, width: 400 }); + const loose = e.create('rect', { + name: 'Loose', + x: 500, + y: 0, + width: 80, + height: 80, + opacity: 0.5, + }); + return { e, frame, group, loose }; +} + +test('a document saves, loads and saves the same text', () => { + const { e } = sampleEditor(); + const saved = e.files.save(); + const other = createEditor(); + other.files.load(saved); + assert.equal(other.files.save(), saved); + const fixture = createEditor(); + fixture.files.load(DOC_V5); + const once = fixture.files.save(); + const again = createEditor(); + again.files.load(once); + assert.equal(again.files.save(), once); +}); + +test('plugin API v1 keeps its members and its shortcut table', () => { + const { e } = sampleEditor(); + let api: any; + e.plugins.load({ + name: 'probe', + capabilities: ['export'], + main: (a) => (api = a), + }); + assert.deepEqual(Object.keys(api).sort(), V1_SURFACE); + assert.deepEqual(JSON.parse(JSON.stringify(api.shortcuts())), V1_SHORTCUTS); +}); diff --git a/benchmarks/proof-build/hidden/_global/globals.json b/benchmarks/proof-build/hidden/_global/globals.json new file mode 100644 index 00000000..635c3e74 --- /dev/null +++ b/benchmarks/proof-build/hidden/_global/globals.json @@ -0,0 +1 @@ +{ "global-rules.test.ts": ["proof-only", "stale"] } diff --git a/benchmarks/proof-build/results/haiku-compare.json b/benchmarks/proof-build/results/haiku-compare.json new file mode 100644 index 00000000..e9a6a809 --- /dev/null +++ b/benchmarks/proof-build/results/haiku-compare.json @@ -0,0 +1,4650 @@ +{ + "work": "", + "runs": ["r1", "r2", "r3"], + "split": "all", + "base": "none", + "flags": { + "brief": { + "sessions": 70, + "excluded": 1, + "left_folder": 1, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 44, + "edited_test_support": 0 + }, + "dump": { + "sessions": 70, + "excluded": 5, + "left_folder": 5, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 42, + "edited_test_support": 0 + }, + "dump10x": { + "sessions": 26, + "excluded": 2, + "left_folder": 2, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 16, + "edited_test_support": 0 + }, + "find": { + "sessions": 70, + "excluded": 4, + "left_folder": 4, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 44, + "edited_test_support": 0 + }, + "findcode": { + "sessions": 70, + "excluded": 5, + "left_folder": 5, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 44, + "edited_test_support": 0 + }, + "none": { + "sessions": 70, + "excluded": 4, + "left_folder": 4, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 43, + "edited_test_support": 0 + }, + "skill": { + "sessions": 70, + "excluded": 1, + "left_folder": 1, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 46, + "edited_test_support": 0 + } + }, + "droppedPairs": [ + "B-P3#r1", + "B-P3#r2", + "B-P3#r3", + "B-P5#r3", + "B-S2#r1", + "B-S2#r3", + "B-S4#r1", + "B-S4#r3", + "B-V1#r2", + "B-V2#r1", + "B-V3#r3", + "B-V4#r3", + "B-V5#r1", + "L-P1#r1", + "L-S1#r1", + "L-V1#r1" + ], + "holm_primary": { + "stale:dump": 0.313, + "stale:brief": 0.5625, + "stale:skill": 1, + "stale:find": 1, + "stale:findcode": 1, + "proof-only:skill": 1, + "proof-only:brief": 1, + "proof-only:find": 1, + "proof-only:findcode": 1, + "proof-only:dump": 1 + }, + "holm_all_green": { + "dump": 0.0585, + "brief": 0.086, + "skill": 0.2109, + "find": 0.2109, + "findcode": 0.2891 + }, + "comparisons": [ + { + "arm": "skill", + "base": "none", + "build": { + "pairs": 41, + "all_green": { + "arm": 29, + "base": 35, + "wins": ["B-P8#r1"], + "losses": [ + "B-S3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "acceptance": { + "arm": 40, + "base": 40, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 32, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "constraints_raw": { + "arm": 33, + "base": 38, + "wins": ["B-P8#r1", "B-V2#r2"], + "losses": [ + "B-S3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.1797, + "taskSignTest": { + "arm": 2, + "base": 4, + "p": 0.6875 + } + }, + "regressions": { + "arm": 38, + "base": 38, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 13, + "unjudged": 0, + "score": { + "arm": 0.898, + "base": 0.919 + }, + "wins": 3, + "losses": 4, + "wilcoxon": { + "n": 7, + "w_plus": 11, + "p": 0.6875 + }, + "avoid_violations": { + "arm": 2, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 3904, + "base": 3705, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 44303, + "base": 36748, + "ratio": 1.232, + "armCheaper": 7 + }, + "read_tokens": { + "arm": 40483, + "base": 33065, + "ratio": 1.256, + "armCheaper": 7 + }, + "processed_tokens": { + "arm": 597670, + "base": 485129, + "ratio": 1.263, + "armCheaper": 11 + }, + "tool_calls": { + "arm": 17, + "base": 16 + }, + "turns": { + "arm": 16, + "base": 14 + }, + "wall_ms": { + "arm": 105534, + "base": 92063, + "ratio": 1.038, + "armCheaper": 19 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 2, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3868, + "base": 3667, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 64750, + "base": 41785, + "ratio": 1.384, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 60882, + "base": 38142, + "ratio": 1.414, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": 406937, + "base": 239226, + "ratio": 1.7, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 12, + "base": 9 + }, + "turns": { + "arm": 9, + "base": 8 + }, + "wall_ms": { + "arm": 112533, + "base": 82972, + "ratio": 1.334, + "armCheaper": 4 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 2, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 10, + "all_green": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 6, + "base": 8, + "wins": ["B-V2#r2"], + "losses": ["B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.625, + "taskSignTest": { + "arm": 1, + "base": 2, + "p": 1 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 0.714, + "base": 0.571 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3904, + "base": 3715, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 60087, + "base": 50224, + "ratio": 1.257, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 56155, + "base": 46531, + "ratio": 1.27, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 937336, + "base": 549351, + "ratio": 1.418, + "armCheaper": 1 + }, + "wall_ms": { + "arm": 134648, + "base": 112090, + "ratio": 1.074, + "armCheaper": 4 + } + } + }, + "control": { + "build": { + "pairs": 9, + "all_green": { + "arm": 6, + "base": 6, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 4, + "unjudged": 0, + "score": { + "arm": 0.929, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3892, + "base": 3705, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 34803, + "base": 29322, + "ratio": 1.222, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 30890, + "base": 25629, + "ratio": 1.265, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 365852, + "base": 310276, + "ratio": 1.188, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 73564, + "base": 72450, + "ratio": 0.966, + "armCheaper": 8 + } + } + }, + "proof-only": { + "build": { + "pairs": 14, + "all_green": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 14, + "base": 14, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 0.887, + "base": 0.875 + }, + "wins": 2, + "losses": 1, + "wilcoxon": { + "n": 3, + "w_plus": 3, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3892, + "base": 3695, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 45248, + "base": 38074, + "ratio": 1.229, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 41430, + "base": 34348, + "ratio": 1.245, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 597670, + "base": 397396, + "ratio": 1.288, + "armCheaper": 5 + }, + "wall_ms": { + "arm": 111427, + "base": 95008, + "ratio": 1.234, + "armCheaper": 7 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S1#r2", "B-S3#r2", "B-S3#r3"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S1#r2", "B-S3#r2", "B-S3#r3"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S1#r2", "B-S3#r2", "B-S3#r3"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 0.938, + "base": 1 + }, + "wins": 0, + "losses": 2, + "wilcoxon": { + "n": 2, + "w_plus": 0, + "p": 0.5 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3868, + "base": 3674, + "ratio": 1.052, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 50866, + "base": 37744, + "ratio": 1.384, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 46985, + "base": 34077, + "ratio": 1.415, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 619977, + "base": 340326, + "ratio": 1.564, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 112533, + "base": 92063, + "ratio": 1.073, + "armCheaper": 4 + } + } + } + } + }, + { + "arm": "brief", + "base": "none", + "build": { + "pairs": 41, + "all_green": { + "arm": 27, + "base": 35, + "wins": ["B-P8#r1"], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0215, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "acceptance": { + "arm": 41, + "base": 40, + "wins": ["B-V2#r2"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 30, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0215, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "constraints_raw": { + "arm": 30, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0215, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "regressions": { + "arm": 38, + "base": 38, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 13, + "unjudged": 0, + "score": { + "arm": 0.963, + "base": 0.919 + }, + "wins": 5, + "losses": 2, + "wilcoxon": { + "n": 7, + "w_plus": 21, + "p": 0.2969 + }, + "avoid_violations": { + "arm": 0, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 15514, + "base": 3705, + "ratio": 4.184, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 56774, + "base": 36748, + "ratio": 1.504, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 41300, + "base": 33065, + "ratio": 1.226, + "armCheaper": 8 + }, + "processed_tokens": { + "arm": 678151, + "base": 485129, + "ratio": 1.552, + "armCheaper": 7 + }, + "tool_calls": { + "arm": 17, + "base": 16 + }, + "turns": { + "arm": 14, + "base": 14 + }, + "wall_ms": { + "arm": 97507, + "base": 92063, + "ratio": 1.05, + "armCheaper": 17 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 15476, + "base": 3667, + "ratio": 4.216, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 62583, + "base": 41785, + "ratio": 1.376, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 47104, + "base": 38142, + "ratio": 1.131, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 377600, + "base": 239226, + "ratio": 1.71, + "armCheaper": 1 + }, + "tool_calls": { + "arm": 10, + "base": 9 + }, + "turns": { + "arm": 8, + "base": 8 + }, + "wall_ms": { + "arm": 85485, + "base": 82972, + "ratio": 1.157, + "armCheaper": 5 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 10, + "all_green": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 10, + "base": 9, + "wins": ["B-V2#r2"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 0.857, + "base": 0.571 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 15522, + "base": 3715, + "ratio": 4.179, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 70530, + "base": 50224, + "ratio": 1.462, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 54996, + "base": 46531, + "ratio": 1.267, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 1175257, + "base": 549351, + "ratio": 1.71, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 144194, + "base": 112090, + "ratio": 1.286, + "armCheaper": 4 + } + } + }, + "control": { + "build": { + "pairs": 9, + "all_green": { + "arm": 6, + "base": 6, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 4, + "unjudged": 0, + "score": { + "arm": 0.964, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 15501, + "base": 3705, + "ratio": 4.187, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 42025, + "base": 29322, + "ratio": 1.638, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 26576, + "base": 25629, + "ratio": 1.251, + "armCheaper": 4 + }, + "processed_tokens": { + "arm": 400876, + "base": 310276, + "ratio": 1.481, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 73675, + "base": 72450, + "ratio": 1.032, + "armCheaper": 6 + } + } + }, + "proof-only": { + "build": { + "pairs": 14, + "all_green": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 14, + "base": 14, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 0.975, + "base": 0.875 + }, + "wins": 4, + "losses": 0, + "wilcoxon": { + "n": 4, + "w_plus": 10, + "p": 0.125 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 15512, + "base": 3695, + "ratio": 4.197, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 57202, + "base": 38074, + "ratio": 1.412, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 41771, + "base": 34348, + "ratio": 1.131, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 623797, + "base": 397396, + "ratio": 1.642, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 100484, + "base": 95008, + "ratio": 1.094, + "armCheaper": 7 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-S1#r1", "B-S3#r1", "B-S1#r2", "B-S3#r2", "B-S3#r3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-S1#r1", "B-S3#r1", "B-S1#r2", "B-S3#r2", "B-S3#r3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-S1#r1", "B-S3#r1", "B-S1#r2", "B-S3#r2", "B-S3#r3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 0.976, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 15473, + "base": 3674, + "ratio": 4.213, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 59212, + "base": 37744, + "ratio": 1.504, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 43697, + "base": 34077, + "ratio": 1.215, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 620543, + "base": 340326, + "ratio": 1.361, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 95457, + "base": 92063, + "ratio": 1.087, + "armCheaper": 5 + } + } + } + } + }, + { + "arm": "find", + "base": "none", + "build": { + "pairs": 41, + "all_green": { + "arm": 29, + "base": 35, + "wins": ["B-P8#r1"], + "losses": [ + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 1, + "base": 3, + "p": 0.625 + } + }, + "acceptance": { + "arm": 41, + "base": 40, + "wins": ["B-V2#r2"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 32, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 1, + "base": 3, + "p": 0.625 + } + }, + "constraints_raw": { + "arm": 32, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 1, + "base": 3, + "p": 0.625 + } + }, + "regressions": { + "arm": 38, + "base": 38, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 13, + "unjudged": 0, + "score": { + "arm": 0.892, + "base": 0.919 + }, + "wins": 2, + "losses": 3, + "wilcoxon": { + "n": 5, + "w_plus": 4, + "p": 0.5 + }, + "avoid_violations": { + "arm": 2, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 3950, + "base": 3705, + "ratio": 1.067, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 40223, + "base": 36748, + "ratio": 1.092, + "armCheaper": 13 + }, + "read_tokens": { + "arm": 36301, + "base": 33065, + "ratio": 1.094, + "armCheaper": 15 + }, + "processed_tokens": { + "arm": 486985, + "base": 485129, + "ratio": 1.149, + "armCheaper": 14 + }, + "tool_calls": { + "arm": 18, + "base": 16 + }, + "turns": { + "arm": 15, + "base": 14 + }, + "wall_ms": { + "arm": 96435, + "base": 92063, + "ratio": 1.137, + "armCheaper": 15 + }, + "proof_reads": { + "arm": 4, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3920, + "base": 3667, + "ratio": 1.07, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 53928, + "base": 41785, + "ratio": 1.15, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 49979, + "base": 38142, + "ratio": 1.157, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 315326, + "base": 239226, + "ratio": 1.218, + "armCheaper": 4 + }, + "tool_calls": { + "arm": 11, + "base": 9 + }, + "turns": { + "arm": 9, + "base": 8 + }, + "wall_ms": { + "arm": 96774, + "base": 82972, + "ratio": 1.195, + "armCheaper": 4 + }, + "proof_reads": { + "arm": 3, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 10, + "all_green": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 10, + "base": 9, + "wins": ["B-V2#r2"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 0.571, + "base": 0.571 + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3958, + "base": 3715, + "ratio": 1.067, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 58223, + "base": 50224, + "ratio": 1.101, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 54242, + "base": 46531, + "ratio": 1.104, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 809882, + "base": 549351, + "ratio": 1.225, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 136846, + "base": 112090, + "ratio": 1.184, + "armCheaper": 4 + } + } + }, + "control": { + "build": { + "pairs": 9, + "all_green": { + "arm": 6, + "base": 6, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 4, + "unjudged": 0, + "score": { + "arm": 0.929, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3941, + "base": 3705, + "ratio": 1.067, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 30179, + "base": 29322, + "ratio": 1.192, + "armCheaper": 5 + }, + "read_tokens": { + "arm": 26245, + "base": 25629, + "ratio": 1.204, + "armCheaper": 5 + }, + "processed_tokens": { + "arm": 344171, + "base": 310276, + "ratio": 1.233, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 77675, + "base": 72450, + "ratio": 1.144, + "armCheaper": 4 + } + } + }, + "proof-only": { + "build": { + "pairs": 14, + "all_green": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 14, + "base": 14, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 0.863, + "base": 0.875 + }, + "wins": 2, + "losses": 2, + "wilcoxon": { + "n": 4, + "w_plus": 4, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3950, + "base": 3695, + "ratio": 1.069, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 44033, + "base": 38074, + "ratio": 1.054, + "armCheaper": 5 + }, + "read_tokens": { + "arm": 40055, + "base": 34348, + "ratio": 1.053, + "armCheaper": 7 + }, + "processed_tokens": { + "arm": 429222, + "base": 397396, + "ratio": 1.031, + "armCheaper": 8 + }, + "wall_ms": { + "arm": 100202, + "base": 95008, + "ratio": 1.137, + "armCheaper": 7 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S3#r2", "B-S3#r3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S3#r2", "B-S3#r3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "constraints_raw": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S3#r2", "B-S3#r3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 1 + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3927, + "base": 3674, + "ratio": 1.068, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 42505, + "base": 37744, + "ratio": 1.112, + "armCheaper": 4 + }, + "read_tokens": { + "arm": 38580, + "base": 34077, + "ratio": 1.118, + "armCheaper": 4 + }, + "processed_tokens": { + "arm": 457071, + "base": 340326, + "ratio": 1.118, + "armCheaper": 4 + }, + "wall_ms": { + "arm": 96435, + "base": 92063, + "ratio": 1.182, + "armCheaper": 4 + } + } + } + } + }, + { + "arm": "findcode", + "base": "none", + "build": { + "pairs": 41, + "all_green": { + "arm": 31, + "base": 35, + "wins": ["B-P8#r1", "B-V2#r2"], + "losses": [ + "B-S3#r1", + "B-V4#r1", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.2891, + "taskSignTest": { + "arm": 2, + "base": 3, + "p": 1 + } + }, + "acceptance": { + "arm": 40, + "base": 40, + "wins": ["B-V2#r2"], + "losses": ["B-V2#r3"], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 34, + "base": 38, + "wins": ["B-P8#r1", "B-V2#r2"], + "losses": [ + "B-S3#r1", + "B-V4#r1", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.2891, + "taskSignTest": { + "arm": 2, + "base": 3, + "p": 1 + } + }, + "constraints_raw": { + "arm": 34, + "base": 38, + "wins": ["B-P8#r1", "B-V2#r2"], + "losses": [ + "B-S3#r1", + "B-V4#r1", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S3#r3" + ], + "p": 0.2891, + "taskSignTest": { + "arm": 2, + "base": 3, + "p": 1 + } + }, + "regressions": { + "arm": 38, + "base": 38, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 13, + "unjudged": 0, + "score": { + "arm": 0.957, + "base": 0.919 + }, + "wins": 2, + "losses": 1, + "wilcoxon": { + "n": 3, + "w_plus": 5, + "p": 0.5 + }, + "avoid_violations": { + "arm": 1, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 4017, + "base": 3705, + "ratio": 1.082, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 47280, + "base": 36748, + "ratio": 1.233, + "armCheaper": 4 + }, + "read_tokens": { + "arm": 43179, + "base": 33065, + "ratio": 1.246, + "armCheaper": 4 + }, + "processed_tokens": { + "arm": 667359, + "base": 485129, + "ratio": 1.244, + "armCheaper": 11 + }, + "tool_calls": { + "arm": 19, + "base": 16 + }, + "turns": { + "arm": 16, + "base": 14 + }, + "wall_ms": { + "arm": 112091, + "base": 92063, + "ratio": 1.315, + "armCheaper": 8 + }, + "proof_reads": { + "arm": 9, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3974, + "base": 3667, + "ratio": 1.085, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 62075, + "base": 41785, + "ratio": 1.308, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 58105, + "base": 38142, + "ratio": 1.335, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": 431903, + "base": 239226, + "ratio": 1.732, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 13, + "base": 9 + }, + "turns": { + "arm": 10, + "base": 8 + }, + "wall_ms": { + "arm": 121556, + "base": 82972, + "ratio": 1.442, + "armCheaper": 1 + }, + "proof_reads": { + "arm": 13, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 10, + "all_green": { + "arm": 6, + "base": 8, + "wins": ["B-V2#r2"], + "losses": ["B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.625, + "taskSignTest": { + "arm": 1, + "base": 2, + "p": 1 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": ["B-V2#r2"], + "losses": ["B-V2#r3"], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 6, + "base": 8, + "wins": ["B-V2#r2"], + "losses": ["B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.625, + "taskSignTest": { + "arm": 1, + "base": 2, + "p": 1 + } + }, + "constraints_raw": { + "arm": 6, + "base": 8, + "wins": ["B-V2#r2"], + "losses": ["B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.625, + "taskSignTest": { + "arm": 1, + "base": 2, + "p": 1 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.571 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 4028, + "base": 3715, + "ratio": 1.082, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 58051, + "base": 50224, + "ratio": 1.224, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 54013, + "base": 46531, + "ratio": 1.236, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": 939461, + "base": 549351, + "ratio": 1.365, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 151965, + "base": 112090, + "ratio": 1.37, + "armCheaper": 2 + } + } + }, + "control": { + "build": { + "pairs": 9, + "all_green": { + "arm": 6, + "base": 6, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 4, + "unjudged": 0, + "score": { + "arm": 1, + "base": 1 + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 4007, + "base": 3705, + "ratio": 1.083, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 35076, + "base": 29322, + "ratio": 1.276, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 31074, + "base": 25629, + "ratio": 1.303, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 391938, + "base": 310276, + "ratio": 1.41, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 84811, + "base": 72450, + "ratio": 1.234, + "armCheaper": 2 + } + } + }, + "proof-only": { + "build": { + "pairs": 14, + "all_green": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 14, + "base": 14, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 0.912, + "base": 0.875 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 4003, + "base": 3695, + "ratio": 1.083, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 53459, + "base": 38074, + "ratio": 1.312, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 49424, + "base": 34348, + "ratio": 1.332, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 667359, + "base": 397396, + "ratio": 1.424, + "armCheaper": 4 + }, + "wall_ms": { + "arm": 125416, + "base": 95008, + "ratio": 1.38, + "armCheaper": 4 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S3#r2", "B-S3#r3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S3#r2", "B-S3#r3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "constraints_raw": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S3#r1", "B-S3#r2", "B-S3#r3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 0.958, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3982, + "base": 3674, + "ratio": 1.082, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 55457, + "base": 37744, + "ratio": 1.303, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 51489, + "base": 34077, + "ratio": 1.318, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 557179, + "base": 340326, + "ratio": 1.548, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 112096, + "base": 92063, + "ratio": 1.281, + "armCheaper": 1 + } + } + } + } + }, + { + "arm": "dump", + "base": "none", + "build": { + "pairs": 41, + "all_green": { + "arm": 26, + "base": 35, + "wins": ["B-P8#r1"], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S1#r3", + "B-S3#r3" + ], + "p": 0.0117, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "acceptance": { + "arm": 41, + "base": 40, + "wins": ["B-V2#r2"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 29, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S1#r3", + "B-S3#r3" + ], + "p": 0.0117, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "constraints_raw": { + "arm": 29, + "base": 38, + "wins": ["B-P8#r1"], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-V3#r1", + "B-V4#r1", + "B-S1#r2", + "B-S3#r2", + "B-V3#r2", + "B-V4#r2", + "B-S1#r3", + "B-S3#r3" + ], + "p": 0.0117, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "regressions": { + "arm": 38, + "base": 38, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 13, + "unjudged": 0, + "score": { + "arm": 0.915, + "base": 0.919 + }, + "wins": 5, + "losses": 3, + "wilcoxon": { + "n": 8, + "w_plus": 17, + "p": 0.9375 + }, + "avoid_violations": { + "arm": 3, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 36279, + "base": 3705, + "ratio": 9.787, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 72118, + "base": 36748, + "ratio": 1.984, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 35761, + "base": 33065, + "ratio": 1.063, + "armCheaper": 14 + }, + "processed_tokens": { + "arm": 962674, + "base": 485129, + "ratio": 1.956, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 19, + "base": 16 + }, + "turns": { + "arm": 14, + "base": 14 + }, + "wall_ms": { + "arm": 95548, + "base": 92063, + "ratio": 1.126, + "armCheaper": 15 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 36241, + "base": 3667, + "ratio": 9.878, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 79425, + "base": 41785, + "ratio": 1.928, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 43212, + "base": 38142, + "ratio": 1.2, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 599378, + "base": 239226, + "ratio": 2.466, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 12, + "base": 9 + }, + "turns": { + "arm": 8, + "base": 8 + }, + "wall_ms": { + "arm": 105421, + "base": 82972, + "ratio": 1.292, + "armCheaper": 0 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 10, + "all_green": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 10, + "base": 9, + "wins": ["B-V2#r2"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1", "B-V3#r2", "B-V4#r2"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 0.714, + "base": 0.571 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 1 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 36285, + "base": 3715, + "ratio": 9.769, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 89125, + "base": 50224, + "ratio": 1.76, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 52826, + "base": 46531, + "ratio": 1.126, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 1432041, + "base": 549351, + "ratio": 1.957, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 138871, + "base": 112090, + "ratio": 1.171, + "armCheaper": 3 + } + } + }, + "control": { + "build": { + "pairs": 9, + "all_green": { + "arm": 6, + "base": 6, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 4, + "unjudged": 0, + "score": { + "arm": 0.964, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 36259, + "base": 3705, + "ratio": 9.795, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 62350, + "base": 29322, + "ratio": 2.137, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 26060, + "base": 25629, + "ratio": 1.057, + "armCheaper": 4 + }, + "processed_tokens": { + "arm": 692277, + "base": 310276, + "ratio": 2.188, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 87399, + "base": 72450, + "ratio": 1.136, + "armCheaper": 4 + } + } + }, + "proof-only": { + "build": { + "pairs": 14, + "all_green": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 14, + "base": 14, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 14, + "base": 13, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.875 + }, + "wins": 4, + "losses": 0, + "wilcoxon": { + "n": 4, + "w_plus": 10, + "p": 0.125 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 36265, + "base": 3695, + "ratio": 9.814, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 70271, + "base": 38074, + "ratio": 1.768, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 34025, + "base": 34348, + "ratio": 1.011, + "armCheaper": 8 + }, + "processed_tokens": { + "arm": 822252, + "base": 397396, + "ratio": 1.807, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 99221, + "base": 95008, + "ratio": 1.141, + "armCheaper": 6 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 2, + "base": 8, + "wins": [], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-S1#r2", + "B-S3#r2", + "B-S1#r3", + "B-S3#r3" + ], + "p": 0.0313, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 2, + "base": 8, + "wins": [], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-S1#r2", + "B-S3#r2", + "B-S1#r3", + "B-S3#r3" + ], + "p": 0.0313, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 2, + "base": 8, + "wins": [], + "losses": [ + "B-S1#r1", + "B-S3#r1", + "B-S1#r2", + "B-S3#r2", + "B-S1#r3", + "B-S3#r3" + ], + "p": 0.0313, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 0.774, + "base": 1 + }, + "wins": 0, + "losses": 2, + "wilcoxon": { + "n": 2, + "w_plus": 0, + "p": 0.5 + }, + "avoid_violations": { + "arm": 2, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 36244, + "base": 3674, + "ratio": 9.862, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 85216, + "base": 37744, + "ratio": 2.003, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 48975, + "base": 34077, + "ratio": 1.214, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 1098169, + "base": 340326, + "ratio": 2.061, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 107042, + "base": 92063, + "ratio": 1.195, + "armCheaper": 2 + } + } + } + } + }, + { + "arm": "dump10x", + "base": "none", + "build": { + "pairs": 13, + "all_green": { + "arm": 8, + "base": 11, + "wins": ["B-P8#r1"], + "losses": ["B-S1#r1", "B-S3#r1", "B-V3#r1", "B-V4#r1"], + "p": 0.375, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "acceptance": { + "arm": 13, + "base": 13, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 12, + "wins": ["B-P8#r1"], + "losses": ["B-S1#r1", "B-S3#r1", "B-V3#r1", "B-V4#r1"], + "p": 0.375, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "constraints_raw": { + "arm": 9, + "base": 12, + "wins": ["B-P8#r1"], + "losses": ["B-S1#r1", "B-S3#r1", "B-V3#r1", "B-V4#r1"], + "p": 0.375, + "taskSignTest": { + "arm": 1, + "base": 4, + "p": 0.375 + } + }, + "regressions": { + "arm": 12, + "base": 12, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 0.896, + "base": 0.975 + }, + "wins": 0, + "losses": 2, + "wilcoxon": { + "n": 2, + "w_plus": 0, + "p": 0.5 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 218715, + "base": 3718, + "ratio": 58.829, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 250096, + "base": 37744, + "ratio": 7.013, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 31419, + "base": 34077, + "ratio": 0.982, + "armCheaper": 7 + }, + "processed_tokens": { + "arm": 3411292, + "base": 524955, + "ratio": 7.805, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 15, + "base": 17 + }, + "turns": { + "arm": 14, + "base": 14 + }, + "wall_ms": { + "arm": 95716, + "base": 89467, + "ratio": 1.012, + "armCheaper": 6 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 218660, + "base": 3666, + "ratio": 59.645, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 262343, + "base": 39071, + "ratio": 6.723, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 43683, + "base": 35425, + "ratio": 1.077, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 1752955, + "base": 226247, + "ratio": 7.564, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 11, + "base": 9 + }, + "turns": { + "arm": 7, + "base": 8 + }, + "wall_ms": { + "arm": 103961, + "base": 80862, + "ratio": 1.353, + "armCheaper": 1 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 3, + "all_green": { + "arm": 1, + "base": 3, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 1, + "base": 3, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 1, + "base": 3, + "wins": [], + "losses": ["B-V3#r1", "B-V4#r1"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": null, + "cost": { + "first_turn_tokens": { + "arm": 218723, + "base": 3724, + "ratio": 58.733, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 268314, + "base": 55484, + "ratio": 4.968, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 49599, + "base": 51760, + "ratio": 0.936, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 4857989, + "base": 549351, + "ratio": 7.805, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 152710, + "base": 112090, + "ratio": 1.362, + "armCheaper": 1 + } + } + }, + "control": { + "build": { + "pairs": 3, + "all_green": { + "arm": 2, + "base": 2, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 0, + "score": { + "arm": 0.929, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 218685, + "base": 3705, + "ratio": 59.024, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 247045, + "base": 30455, + "ratio": 7.813, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 28360, + "base": 26735, + "ratio": 1.217, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 2712375, + "base": 310276, + "ratio": 8.742, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 80483, + "base": 80862, + "ratio": 0.996, + "armCheaper": 3 + } + } + }, + "proof-only": { + "build": { + "pairs": 5, + "all_green": { + "arm": 5, + "base": 4, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 5, + "base": 5, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 4, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 5, + "base": 4, + "wins": ["B-P8#r1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 0, + "score": { + "arm": 0.938, + "base": 0.938 + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 218692, + "base": 3695, + "ratio": 59.186, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 259650, + "base": 38074, + "ratio": 7.037, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 40924, + "base": 34348, + "ratio": 1.049, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 2653573, + "base": 397396, + "ratio": 6.999, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 106840, + "base": 92127, + "ratio": 1.353, + "armCheaper": 3 + } + } + }, + "stale": { + "build": { + "pairs": 2, + "all_green": { + "arm": 0, + "base": 2, + "wins": [], + "losses": ["B-S1#r1", "B-S3#r1"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 2, + "base": 2, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 0, + "base": 2, + "wins": [], + "losses": ["B-S1#r1", "B-S3#r1"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 0, + "base": 2, + "wins": [], + "losses": ["B-S1#r1", "B-S3#r1"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 0.75, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 218675, + "base": 3667, + "ratio": 59.633, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 256458, + "base": 37744, + "ratio": 6.795, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 37783, + "base": 34077, + "ratio": 0.982, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 2953178, + "base": 340326, + "ratio": 5.561, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 95716, + "base": 85151, + "ratio": 1.123, + "armCheaper": 0 + } + } + } + } + } + ], + "dump10x_vs_dump": { + "arm": "dump10x", + "base": "dump", + "build": { + "pairs": 13, + "all_green": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 13, + "base": 13, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 9, + "base": 9, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "regressions": { + "arm": 12, + "base": 12, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 0, + "score": { + "arm": 0.896, + "base": 0.971 + }, + "wins": 0, + "losses": 2, + "wilcoxon": { + "n": 2, + "w_plus": 0, + "p": 0.5 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 218715, + "base": 36285, + "ratio": 6.028, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 250096, + "base": 69452, + "ratio": 3.569, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 31419, + "base": 33187, + "ratio": 0.88, + "armCheaper": 9 + }, + "processed_tokens": { + "arm": 3411292, + "base": 1123238, + "ratio": 3.476, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 15, + "base": 19 + }, + "turns": { + "arm": 14, + "base": 16 + }, + "wall_ms": { + "arm": 95716, + "base": 101759, + "ratio": 0.88, + "armCheaper": 9 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 1 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 218660, + "base": 36237, + "ratio": 6.034, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 262343, + "base": 77948, + "ratio": 3.343, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 43683, + "base": 41722, + "ratio": 1.02, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 1752955, + "base": 611014, + "ratio": 3.529, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 11, + "base": 12 + }, + "turns": { + "arm": 7, + "base": 9 + }, + "wall_ms": { + "arm": 103961, + "base": 93518, + "ratio": 1.112, + "armCheaper": 2 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 1 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 3, + "all_green": { + "arm": 1, + "base": 1, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 1, + "base": 1, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 1, + "base": 1, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": null, + "cost": { + "first_turn_tokens": { + "arm": 218723, + "base": 36292, + "ratio": 6.027, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 268314, + "base": 105315, + "ratio": 2.548, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 49599, + "base": 69023, + "ratio": 0.719, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 4857989, + "base": 1867079, + "ratio": 2.297, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 152710, + "base": 178511, + "ratio": 0.855, + "armCheaper": 3 + } + } + }, + "control": { + "build": { + "pairs": 3, + "all_green": { + "arm": 2, + "base": 2, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 3, + "base": 3, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 0, + "score": { + "arm": 0.929, + "base": 0.929 + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 218685, + "base": 36259, + "ratio": 6.031, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 247045, + "base": 63910, + "ratio": 3.818, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 28360, + "base": 27615, + "ratio": 0.915, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 2712375, + "base": 637520, + "ratio": 3.491, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 80483, + "base": 91259, + "ratio": 0.882, + "armCheaper": 3 + } + } + }, + "proof-only": { + "build": { + "pairs": 5, + "all_green": { + "arm": 5, + "base": 5, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 5, + "base": 5, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 5, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 5, + "base": 5, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 0, + "score": { + "arm": 0.938, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 218692, + "base": 36265, + "ratio": 6.03, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 259650, + "base": 69452, + "ratio": 3.591, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 40924, + "base": 33187, + "ratio": 1.145, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 2653573, + "base": 763432, + "ratio": 3.854, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 106840, + "base": 105752, + "ratio": 1.154, + "armCheaper": 3 + } + } + }, + "stale": { + "build": { + "pairs": 2, + "all_green": { + "arm": 0, + "base": 0, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 2, + "base": 2, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 0, + "base": 0, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 0, + "base": 0, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 1, + "unjudged": 0, + "score": { + "arm": 0.75, + "base": 1 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 218675, + "base": 36246, + "ratio": 6.033, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 256458, + "base": 77948, + "ratio": 3.351, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 37783, + "base": 41722, + "ratio": 0.947, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 2953178, + "base": 1234022, + "ratio": 2.485, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 95716, + "base": 101759, + "ratio": 0.847, + "armCheaper": 2 + } + } + } + } + } +} diff --git a/benchmarks/proof-build/results/haiku-sessions.json b/benchmarks/proof-build/results/haiku-sessions.json new file mode 100644 index 00000000..257c3750 --- /dev/null +++ b/benchmarks/proof-build/results/haiku-sessions.json @@ -0,0 +1,10715 @@ +[ + { + "run": "pilot", + "arm": "brief", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 22, + "context_tokens": 49727, + "processed_tokens": 703577, + "wall_ms": 97941, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "brief", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 24, + "context_tokens": 52449, + "processed_tokens": 578630, + "wall_ms": 100426, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 43234, + "processed_tokens": 507144, + "wall_ms": 63655, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 21, + "context_tokens": 95480, + "processed_tokens": 1236690, + "wall_ms": 155924, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "brief", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 16, + "context_tokens": 85325, + "processed_tokens": 613716, + "wall_ms": 181057, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 24, + "context_tokens": 62934, + "processed_tokens": 909987, + "wall_ms": 91373, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 20, + "context_tokens": 78509, + "processed_tokens": 918577, + "wall_ms": 103815, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 18, + "context_tokens": 75637, + "processed_tokens": 1139932, + "wall_ms": 85857, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 31, + "context_tokens": 103546, + "processed_tokens": 1940989, + "wall_ms": 169425, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 14, + "context_tokens": 69980, + "processed_tokens": 350149, + "wall_ms": 132841, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 27843, + "processed_tokens": 262317, + "wall_ms": 98318, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 28735, + "processed_tokens": 286168, + "wall_ms": 90307, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 38432, + "processed_tokens": 379025, + "wall_ms": 75208, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 24, + "context_tokens": 77863, + "processed_tokens": 1241470, + "wall_ms": 175763, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 48120, + "processed_tokens": 264361, + "wall_ms": 100095, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 15, + "context_tokens": 37936, + "processed_tokens": 530379, + "wall_ms": 139432, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 39113, + "processed_tokens": 432223, + "wall_ms": 115796, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 19, + "context_tokens": 44301, + "processed_tokens": 611674, + "wall_ms": 109314, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 28, + "context_tokens": 71824, + "processed_tokens": 1194809, + "wall_ms": 183259, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 11, + "context_tokens": 58330, + "processed_tokens": 243172, + "wall_ms": 113147, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 18, + "context_tokens": 35321, + "processed_tokens": 381470, + "wall_ms": 79178, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 19, + "context_tokens": 39827, + "processed_tokens": 433794, + "wall_ms": 81420, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 32134, + "processed_tokens": 325423, + "wall_ms": 60799, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 59686, + "processed_tokens": 814862, + "wall_ms": 143130, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 11, + "context_tokens": 59295, + "processed_tokens": 336075, + "wall_ms": 120874, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 21, + "context_tokens": 36636, + "processed_tokens": 413775, + "wall_ms": 56348, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 34278, + "processed_tokens": 336430, + "wall_ms": 73962, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 39923, + "processed_tokens": 491789, + "wall_ms": 73087, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 23, + "tool_calls": 30, + "context_tokens": 68405, + "processed_tokens": 1250305, + "wall_ms": 182578, + "model": "claude-haiku-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 63077, + "processed_tokens": 601023, + "wall_ms": 134714, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 44651, + "processed_tokens": 474079, + "wall_ms": 78728, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 37219, + "processed_tokens": 420044, + "wall_ms": 57014, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 34609, + "processed_tokens": 377546, + "wall_ms": 60814, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 54380, + "processed_tokens": 644229, + "wall_ms": 84661, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 30, + "context_tokens": 75586, + "processed_tokens": 1079491, + "wall_ms": 132921, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 25, + "tool_calls": 26, + "context_tokens": 62841, + "processed_tokens": 1415844, + "wall_ms": 114545, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 13, + "tool_calls": 16, + "context_tokens": 50011, + "processed_tokens": 589372, + "wall_ms": 101155, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 61156, + "processed_tokens": 1189270, + "wall_ms": 100880, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 16, + "context_tokens": 46991, + "processed_tokens": 758097, + "wall_ms": 91025, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 56743, + "processed_tokens": 599709, + "wall_ms": 95457, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 28, + "context_tokens": 59148, + "processed_tokens": 1037061, + "wall_ms": 116218, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 56774, + "processed_tokens": 640104, + "wall_ms": 82010, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 72595, + "processed_tokens": 865530, + "wall_ms": 128960, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 65842, + "processed_tokens": 1141751, + "wall_ms": 150405, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 16, + "tool_calls": 16, + "context_tokens": 64088, + "processed_tokens": 900311, + "wall_ms": 98284, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 17, + "tool_calls": 18, + "context_tokens": 83879, + "processed_tokens": 1176238, + "wall_ms": 144194, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 23, + "context_tokens": 81125, + "processed_tokens": 1267126, + "wall_ms": 153349, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /wt-p3r/node_modules/.bin"], + "proof_reads": 1, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 14, + "context_tokens": 53233, + "processed_tokens": 480195, + "wall_ms": 95125, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 46641, + "processed_tokens": 329019, + "wall_ms": 73675, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 62531, + "processed_tokens": 400876, + "wall_ms": 80103, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 65315, + "processed_tokens": 404653, + "wall_ms": 122083, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 62583, + "processed_tokens": 343601, + "wall_ms": 107213, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 14, + "context_tokens": 82127, + "processed_tokens": 598863, + "wall_ms": 155023, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 65793, + "processed_tokens": 430766, + "wall_ms": 114600, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 63884, + "processed_tokens": 373445, + "wall_ms": 83409, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "brief", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 58790, + "processed_tokens": 457490, + "wall_ms": 103980, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 53947, + "processed_tokens": 541234, + "wall_ms": 66824, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 63910, + "processed_tokens": 693782, + "wall_ms": 57827, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 58660, + "processed_tokens": 815877, + "wall_ms": 92525, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 19, + "context_tokens": 58764, + "processed_tokens": 822252, + "wall_ms": 84975, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 25, + "tool_calls": 35, + "context_tokens": 98803, + "processed_tokens": 2183846, + "wall_ms": 188765, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 22, + "context_tokens": 69452, + "processed_tokens": 1320286, + "wall_ms": 133222, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 19, + "context_tokens": 58773, + "processed_tokens": 763432, + "wall_ms": 84790, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 26, + "context_tokens": 78270, + "processed_tokens": 1123238, + "wall_ms": 122859, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 11, + "context_tokens": 66318, + "processed_tokens": 649255, + "wall_ms": 78630, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 19, + "context_tokens": 88052, + "processed_tokens": 1234022, + "wall_ms": 127298, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 22, + "context_tokens": 76329, + "processed_tokens": 1021947, + "wall_ms": 86391, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 24, + "context_tokens": 76143, + "processed_tokens": 1320291, + "wall_ms": 101759, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 22, + "tool_calls": 22, + "context_tokens": 97950, + "processed_tokens": 1944294, + "wall_ms": 169081, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 89125, + "processed_tokens": 1661212, + "wall_ms": 144836, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": ["acceptance: anything but a positive number is refused"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 95684, + "processed_tokens": 1428373, + "wall_ms": 132846, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 24, + "tool_calls": 24, + "context_tokens": 105315, + "processed_tokens": 2258740, + "wall_ms": 178511, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 31, + "context_tokens": 111382, + "processed_tokens": 1867079, + "wall_ms": 196296, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 20, + "context_tokens": 77680, + "processed_tokens": 1029211, + "wall_ms": 93436, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 10, + "context_tokens": 75145, + "processed_tokens": 611014, + "wall_ms": 92568, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 12, + "context_tokens": 79425, + "processed_tokens": 637520, + "wall_ms": 91259, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": true, + "left_folder_evidence": ["Read: /src/persist/migrations/v4-v5.ts"], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 23, + "context_tokens": 77188, + "processed_tokens": 498174, + "wall_ms": 145522, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 12, + "context_tokens": 70271, + "processed_tokens": 450458, + "wall_ms": 105752, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 22, + "context_tokens": 94212, + "processed_tokens": 656268, + "wall_ms": 157131, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 19, + "context_tokens": 100609, + "processed_tokens": 924458, + "wall_ms": 166266, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 77948, + "processed_tokens": 456917, + "wall_ms": 93518, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": true, + "left_folder_evidence": ["Read: /src/commands/types.ts"], + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 21, + "context_tokens": 81704, + "processed_tokens": 700515, + "wall_ms": 117609, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 247045, + "processed_tokens": 2712375, + "wall_ms": 67798, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 14, + "tool_calls": 15, + "context_tokens": 243990, + "processed_tokens": 3411292, + "wall_ms": 83374, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 237952, + "processed_tokens": 2848106, + "wall_ms": 77591, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 15, + "context_tokens": 259650, + "processed_tokens": 3789431, + "wall_ms": 105865, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 18, + "context_tokens": 270642, + "processed_tokens": 4684749, + "wall_ms": 137845, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 18, + "context_tokens": 247887, + "processed_tokens": 4179622, + "wall_ms": 106840, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 17, + "context_tokens": 244462, + "processed_tokens": 2653573, + "wall_ms": 76206, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 24, + "tool_calls": 24, + "context_tokens": 267931, + "processed_tokens": 6197398, + "wall_ms": 142779, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 238146, + "processed_tokens": 2613352, + "wall_ms": 69213, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 250096, + "processed_tokens": 2953178, + "wall_ms": 95716, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 23, + "context_tokens": 256420, + "processed_tokens": 4018529, + "wall_ms": 128899, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 256458, + "processed_tokens": 3281023, + "wall_ms": 86179, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: cd /tmp"], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 18, + "context_tokens": 264397, + "processed_tokens": 4811322, + "wall_ms": 180401, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 251659, + "processed_tokens": 5186097, + "wall_ms": 132936, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 251239, + "processed_tokens": 3713973, + "wall_ms": 109167, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 18, + "context_tokens": 268314, + "processed_tokens": 4857989, + "wall_ms": 152710, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 21, + "context_tokens": 282135, + "processed_tokens": 4287771, + "wall_ms": 153276, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 20, + "context_tokens": 258085, + "processed_tokens": 3774461, + "wall_ms": 104255, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 7, + "tool_calls": 13, + "context_tokens": 251202, + "processed_tokens": 1711291, + "wall_ms": 80541, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 11, + "context_tokens": 262693, + "processed_tokens": 2012658, + "wall_ms": 80483, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 293675, + "processed_tokens": 1802969, + "wall_ms": 201159, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 262343, + "processed_tokens": 1735890, + "wall_ms": 131441, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 11, + "context_tokens": 280443, + "processed_tokens": 2316275, + "wall_ms": 181309, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.2857, + "excluded": true, + "left_folder_evidence": ["Grep: /test"], + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 30, + "context_tokens": 279665, + "processed_tokens": 2853539, + "wall_ms": 216188, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 261225, + "processed_tokens": 1752955, + "wall_ms": 103961, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "dump10x", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 257632, + "processed_tokens": 1950546, + "wall_ms": 103056, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 35437, + "processed_tokens": 443430, + "wall_ms": 94103, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 1, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 21447, + "processed_tokens": 219781, + "wall_ms": 47851, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 17, + "context_tokens": 26001, + "processed_tokens": 426378, + "wall_ms": 90951, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 24, + "context_tokens": 38646, + "processed_tokens": 584240, + "wall_ms": 113797, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 19, + "context_tokens": 49380, + "processed_tokens": 755488, + "wall_ms": 120917, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 23, + "tool_calls": 29, + "context_tokens": 44675, + "processed_tokens": 873628, + "wall_ms": 146170, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 18, + "context_tokens": 35891, + "processed_tokens": 379218, + "wall_ms": 80635, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 44033, + "processed_tokens": 524989, + "wall_ms": 111914, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 16, + "context_tokens": 44404, + "processed_tokens": 392890, + "wall_ms": 104715, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 47651, + "processed_tokens": 572432, + "wall_ms": 86664, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: cd /tmp"], + "proof_reads": 6, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 24, + "context_tokens": 46968, + "processed_tokens": 532003, + "wall_ms": 119739, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 33715, + "processed_tokens": 387240, + "wall_ms": 72642, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 28, + "tool_calls": 29, + "context_tokens": 58387, + "processed_tokens": 1291163, + "wall_ms": 191817, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 58223, + "processed_tokens": 714609, + "wall_ms": 144228, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": true, + "left_folder_evidence": ["Bash: /tmp"], + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 21, + "context_tokens": 49002, + "processed_tokens": 752406, + "wall_ms": 123452, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 34, + "context_tokens": 69064, + "processed_tokens": 1161143, + "wall_ms": 158705, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 21, + "context_tokens": 61083, + "processed_tokens": 708704, + "wall_ms": 146200, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 49551, + "processed_tokens": 532760, + "wall_ms": 103360, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 26094, + "processed_tokens": 173936, + "wall_ms": 57818, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 9, + "tool_calls": 12, + "context_tokens": 46569, + "processed_tokens": 344171, + "wall_ms": 80913, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 49626, + "processed_tokens": 324756, + "wall_ms": 113322, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 9, + "tool_calls": 10, + "context_tokens": 52547, + "processed_tokens": 357458, + "wall_ms": 96774, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.6875, + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 53928, + "processed_tokens": 277681, + "wall_ms": 105811, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 56082, + "processed_tokens": 431167, + "wall_ms": 115521, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 14, + "context_tokens": 56713, + "processed_tokens": 315326, + "wall_ms": 110622, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "find", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 42790, + "processed_tokens": 281388, + "wall_ms": 82561, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 35591, + "processed_tokens": 491071, + "wall_ms": 96230, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 28013, + "processed_tokens": 387727, + "wall_ms": 81918, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 27520, + "processed_tokens": 367287, + "wall_ms": 89698, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 43731, + "processed_tokens": 488347, + "wall_ms": 106064, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /tmp/"], + "proof_reads": 14, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 26, + "tool_calls": 33, + "context_tokens": 66269, + "processed_tokens": 1341211, + "wall_ms": 179644, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 58379, + "processed_tokens": 1082851, + "wall_ms": 164659, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 45465, + "processed_tokens": 698644, + "wall_ms": 125416, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 20, + "context_tokens": 53459, + "processed_tokens": 810448, + "wall_ms": 136414, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 16, + "context_tokens": 39454, + "processed_tokens": 339300, + "wall_ms": 100661, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 55457, + "processed_tokens": 526840, + "wall_ms": 110327, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 52540, + "processed_tokens": 769078, + "wall_ms": 158866, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 39241, + "processed_tokens": 557179, + "wall_ms": 104708, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 28, + "context_tokens": 72003, + "processed_tokens": 1132675, + "wall_ms": 205897, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 27, + "tool_calls": 26, + "context_tokens": 58051, + "processed_tokens": 1266457, + "wall_ms": 168669, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": true, + "left_folder_evidence": [ + "Bash: /wt-p3r/node_modules/.bin/prettier", + "Bash: /wt-p3r/node_modules/.bin/prettier", + "Bash: /wt-p3r/node_modules/.bin/prettier", + "Bash: /wt-p3r/node_modules/.bin/prettier" + ], + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 66291, + "processed_tokens": 1116469, + "wall_ms": 168632, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 14, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 28, + "tool_calls": 32, + "context_tokens": 70068, + "processed_tokens": 1474691, + "wall_ms": 200161, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 27, + "context_tokens": 68017, + "processed_tokens": 1012346, + "wall_ms": 192604, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 21, + "context_tokens": 47467, + "processed_tokens": 491199, + "wall_ms": 118203, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 12, + "context_tokens": 44317, + "processed_tokens": 391938, + "wall_ms": 99744, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 44624, + "processed_tokens": 227772, + "wall_ms": 69567, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 55568, + "processed_tokens": 315757, + "wall_ms": 88162, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 17, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 14, + "context_tokens": 68309, + "processed_tokens": 486835, + "wall_ms": 124710, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 16, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 16, + "context_tokens": 75222, + "processed_tokens": 507517, + "wall_ms": 160061, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 20, + "context_tokens": 69953, + "processed_tokens": 794140, + "wall_ms": 171998, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 14, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 16, + "context_tokens": 57436, + "processed_tokens": 663139, + "wall_ms": 136213, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "findcode", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 56605, + "processed_tokens": 482896, + "wall_ms": 143150, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 27014, + "processed_tokens": 310276, + "wall_ms": 89467, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 22718, + "processed_tokens": 321195, + "wall_ms": 72450, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 30455, + "processed_tokens": 424273, + "wall_ms": 83213, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 35405, + "processed_tokens": 392455, + "wall_ms": 76872, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 29, + "context_tokens": 58496, + "processed_tokens": 859157, + "wall_ms": 145035, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 44511, + "processed_tokens": 760387, + "wall_ms": 109087, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 29, + "context_tokens": 45082, + "processed_tokens": 733856, + "wall_ms": 132624, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 38074, + "processed_tokens": 524955, + "wall_ms": 87511, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the fix saves as a newer version, reached by a step from version 5" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 16, + "context_tokens": 33421, + "processed_tokens": 373410, + "wall_ms": 92127, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 35661, + "processed_tokens": 340326, + "wall_ms": 57775, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 25, + "context_tokens": 38718, + "processed_tokens": 663468, + "wall_ms": 129218, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 20, + "context_tokens": 37744, + "processed_tokens": 692139, + "wall_ms": 85151, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 46374, + "processed_tokens": 560210, + "wall_ms": 146762, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 24, + "tool_calls": 23, + "context_tokens": 50659, + "processed_tokens": 995231, + "wall_ms": 158990, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 50418, + "processed_tokens": 694332, + "wall_ms": 106936, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 16, + "context_tokens": 56701, + "processed_tokens": 547932, + "wall_ms": 112090, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 55484, + "processed_tokens": 549351, + "wall_ms": 108003, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": [ + "Bash: /p4/orig.ts", + "Bash: cd /p4" + ], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 25, + "tool_calls": 24, + "context_tokens": 49759, + "processed_tokens": 1047357, + "wall_ms": 147634, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 33884, + "processed_tokens": 226247, + "wall_ms": 80862, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 39071, + "processed_tokens": 225437, + "wall_ms": 50500, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 9, + "context_tokens": 49629, + "processed_tokens": 291002, + "wall_ms": 96393, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 7, + "context_tokens": 31526, + "processed_tokens": 149513, + "wall_ms": 57663, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 14, + "context_tokens": 62579, + "processed_tokens": 397396, + "wall_ms": 133983, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 59248, + "processed_tokens": 410078, + "wall_ms": 102334, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 49304, + "processed_tokens": 315215, + "wall_ms": 92565, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "none", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.5714, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 15, + "context_tokens": 33135, + "processed_tokens": 176458, + "wall_ms": 70493, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 16, + "tool_calls": 15, + "context_tokens": 32193, + "processed_tokens": 497105, + "wall_ms": 72306, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 28444, + "processed_tokens": 374180, + "wall_ms": 68793, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 35144, + "processed_tokens": 363550, + "wall_ms": 67884, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 15, + "tool_calls": 19, + "context_tokens": 43602, + "processed_tokens": 597670, + "wall_ms": 105534, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 26, + "tool_calls": 31, + "context_tokens": 67703, + "processed_tokens": 1383962, + "wall_ms": 169972, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 20, + "tool_calls": 24, + "context_tokens": 54693, + "processed_tokens": 929619, + "wall_ms": 117192, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 17, + "tool_calls": 24, + "context_tokens": 44303, + "processed_tokens": 678215, + "wall_ms": 118123, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 17, + "tool_calls": 16, + "context_tokens": 52674, + "processed_tokens": 712921, + "wall_ms": 107946, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 15, + "context_tokens": 39793, + "processed_tokens": 481029, + "wall_ms": 104548, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 49823, + "processed_tokens": 633360, + "wall_ms": 89406, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 16, + "tool_calls": 22, + "context_tokens": 49900, + "processed_tokens": 662268, + "wall_ms": 118159, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 12, + "context_tokens": 36615, + "processed_tokens": 363539, + "wall_ms": 61716, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 24, + "tool_calls": 37, + "context_tokens": 75033, + "processed_tokens": 1423658, + "wall_ms": 180442, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 18, + "context_tokens": 49630, + "processed_tokens": 804157, + "wall_ms": 112327, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 50583, + "processed_tokens": 848651, + "wall_ms": 122529, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 70931, + "processed_tokens": 1115880, + "wall_ms": 176703, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 21, + "tool_calls": 26, + "context_tokens": 71790, + "processed_tokens": 1162141, + "wall_ms": 194681, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 18, + "context_tokens": 47866, + "processed_tokens": 518895, + "wall_ms": 98686, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 37941, + "processed_tokens": 266562, + "wall_ms": 78022, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 50301, + "processed_tokens": 383311, + "wall_ms": 85396, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 9, + "tool_calls": 12, + "context_tokens": 68144, + "processed_tokens": 418704, + "wall_ms": 115466, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 17, + "context_tokens": 64750, + "processed_tokens": 616588, + "wall_ms": 146217, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.6875, + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 74047, + "processed_tokens": 441097, + "wall_ms": 111330, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.9286, + "excluded": false, + "proof_reads": 1, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 67216, + "processed_tokens": 576071, + "wall_ms": 124179, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.9375, + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 71544, + "processed_tokens": 619977, + "wall_ms": 123505, + "model": "claude-haiku-5-5" + }, + { + "run": "r1", + "arm": "skill", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 52738, + "processed_tokens": 422722, + "wall_ms": 98952, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 23, + "context_tokens": 48111, + "processed_tokens": 678151, + "wall_ms": 97507, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 41593, + "processed_tokens": 441981, + "wall_ms": 66805, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 12, + "context_tokens": 36885, + "processed_tokens": 471258, + "wall_ms": 73813, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 52735, + "processed_tokens": 614133, + "wall_ms": 79345, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 23, + "tool_calls": 32, + "context_tokens": 83878, + "processed_tokens": 1586288, + "wall_ms": 146129, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 26, + "tool_calls": 27, + "context_tokens": 69321, + "processed_tokens": 1603465, + "wall_ms": 141541, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 16, + "tool_calls": 22, + "context_tokens": 57202, + "processed_tokens": 796865, + "wall_ms": 89212, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 26, + "context_tokens": 69740, + "processed_tokens": 1181450, + "wall_ms": 140846, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 49908, + "processed_tokens": 533378, + "wall_ms": 84680, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 18, + "tool_calls": 22, + "context_tokens": 62614, + "processed_tokens": 966163, + "wall_ms": 100610, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 23, + "context_tokens": 63027, + "processed_tokens": 678922, + "wall_ms": 101580, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 51286, + "processed_tokens": 577482, + "wall_ms": 75284, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 17, + "context_tokens": 59212, + "processed_tokens": 939057, + "wall_ms": 132081, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 62934, + "processed_tokens": 1207847, + "wall_ms": 130311, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 15, + "context_tokens": 59027, + "processed_tokens": 843620, + "wall_ms": 117724, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 26, + "tool_calls": 25, + "context_tokens": 89438, + "processed_tokens": 1862118, + "wall_ms": 175809, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 26, + "context_tokens": 81273, + "processed_tokens": 1387894, + "wall_ms": 137401, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 18, + "tool_calls": 23, + "context_tokens": 63646, + "processed_tokens": 1028699, + "wall_ms": 137484, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 55621, + "processed_tokens": 305838, + "wall_ms": 85485, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 6, + "tool_calls": 7, + "context_tokens": 42025, + "processed_tokens": 228078, + "wall_ms": 56970, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 71376, + "processed_tokens": 694652, + "wall_ms": 134053, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 56756, + "processed_tokens": 335160, + "wall_ms": 84985, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 64512, + "processed_tokens": 408110, + "wall_ms": 96026, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.9286, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 78720, + "processed_tokens": 678398, + "wall_ms": 153279, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 66944, + "processed_tokens": 551289, + "wall_ms": 103178, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "brief", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 53635, + "processed_tokens": 377600, + "wall_ms": 76271, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 59877, + "processed_tokens": 826053, + "wall_ms": 87399, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 63369, + "processed_tokens": 692277, + "wall_ms": 63386, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 62350, + "processed_tokens": 892376, + "wall_ms": 94890, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 76998, + "processed_tokens": 930737, + "wall_ms": 77374, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 23, + "tool_calls": 33, + "context_tokens": 105580, + "processed_tokens": 1953913, + "wall_ms": 155898, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 19, + "context_tokens": 70183, + "processed_tokens": 969549, + "wall_ms": 99221, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 69673, + "processed_tokens": 669536, + "wall_ms": 85142, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 21, + "context_tokens": 84814, + "processed_tokens": 1174883, + "wall_ms": 115862, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 72118, + "processed_tokens": 808255, + "wall_ms": 111816, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 16, + "context_tokens": 87440, + "processed_tokens": 1204444, + "wall_ms": 107042, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 20, + "context_tokens": 82191, + "processed_tokens": 1098169, + "wall_ms": 107227, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 61698, + "processed_tokens": 947376, + "wall_ms": 81976, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 32, + "context_tokens": 97411, + "processed_tokens": 1729056, + "wall_ms": 197057, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /wt-p3r/node_modules/.bin"], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 17, + "context_tokens": 70071, + "processed_tokens": 717126, + "wall_ms": 114676, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 89886, + "processed_tokens": 1707920, + "wall_ms": 146059, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 100900, + "processed_tokens": 1835259, + "wall_ms": 178757, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 23, + "context_tokens": 91976, + "processed_tokens": 1432041, + "wall_ms": 138871, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 16, + "context_tokens": 64665, + "processed_tokens": 681926, + "wall_ms": 91886, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 13, + "context_tokens": 71676, + "processed_tokens": 413521, + "wall_ms": 94044, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 21, + "context_tokens": 69794, + "processed_tokens": 430069, + "wall_ms": 93734, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 9, + "tool_calls": 10, + "context_tokens": 93064, + "processed_tokens": 642394, + "wall_ms": 204167, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 8, + "context_tokens": 55617, + "processed_tokens": 268744, + "wall_ms": 79790, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 14, + "tool_calls": 18, + "context_tokens": 131925, + "processed_tokens": 1277630, + "wall_ms": 238326, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.5714, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 9, + "tool_calls": 13, + "context_tokens": 99773, + "processed_tokens": 693961, + "wall_ms": 148151, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 85896, + "processed_tokens": 599378, + "wall_ms": 105421, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "dump", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 17, + "context_tokens": 81930, + "processed_tokens": 544518, + "wall_ms": 114039, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 22, + "context_tokens": 42648, + "processed_tokens": 614668, + "wall_ms": 110325, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 33622, + "processed_tokens": 398610, + "wall_ms": 77675, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 14, + "context_tokens": 27628, + "processed_tokens": 342884, + "wall_ms": 75136, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 36886, + "processed_tokens": 458479, + "wall_ms": 79017, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /tmp"], + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 61755, + "processed_tokens": 1021060, + "wall_ms": 125770, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 20, + "context_tokens": 49252, + "processed_tokens": 738348, + "wall_ms": 100202, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 21, + "context_tokens": 52085, + "processed_tokens": 835666, + "wall_ms": 117926, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 23, + "context_tokens": 54966, + "processed_tokens": 833288, + "wall_ms": 140008, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 18, + "context_tokens": 38150, + "processed_tokens": 476829, + "wall_ms": 90298, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 36497, + "processed_tokens": 359270, + "wall_ms": 66571, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 24, + "context_tokens": 35140, + "processed_tokens": 496220, + "wall_ms": 96435, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 14, + "context_tokens": 32465, + "processed_tokens": 297672, + "wall_ms": 66952, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 31, + "context_tokens": 54810, + "processed_tokens": 937887, + "wall_ms": 165656, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 19, + "context_tokens": 46349, + "processed_tokens": 718503, + "wall_ms": 119359, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 47802, + "processed_tokens": 809882, + "wall_ms": 107935, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 27, + "context_tokens": 79809, + "processed_tokens": 1170939, + "wall_ms": 181575, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 24, + "context_tokens": 68988, + "processed_tokens": 1046466, + "wall_ms": 174652, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 18, + "context_tokens": 40223, + "processed_tokens": 564543, + "wall_ms": 93836, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 6, + "tool_calls": 8, + "context_tokens": 38839, + "processed_tokens": 178637, + "wall_ms": 69607, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 18, + "context_tokens": 58906, + "processed_tokens": 425369, + "wall_ms": 88337, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 55132, + "processed_tokens": 384531, + "wall_ms": 138519, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 39790, + "processed_tokens": 246901, + "wall_ms": 83190, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 11, + "context_tokens": 61612, + "processed_tokens": 281873, + "wall_ms": 121075, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 67847, + "processed_tokens": 410199, + "wall_ms": 137049, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 14, + "context_tokens": 67754, + "processed_tokens": 457071, + "wall_ms": 129945, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "find", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.5714, + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 6, + "tool_calls": 7, + "context_tokens": 36822, + "processed_tokens": 163889, + "wall_ms": 75562, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 37407, + "processed_tokens": 512463, + "wall_ms": 67710, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 32657, + "processed_tokens": 363544, + "wall_ms": 86726, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 21, + "context_tokens": 32830, + "processed_tokens": 647327, + "wall_ms": 83164, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 17, + "context_tokens": 34198, + "processed_tokens": 509996, + "wall_ms": 81330, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 29, + "context_tokens": 83095, + "processed_tokens": 1035718, + "wall_ms": 163074, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 26, + "tool_calls": 27, + "context_tokens": 51149, + "processed_tokens": 1116855, + "wall_ms": 189329, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 23, + "context_tokens": 46406, + "processed_tokens": 731854, + "wall_ms": 133597, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 18, + "context_tokens": 53997, + "processed_tokens": 791249, + "wall_ms": 123173, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 21, + "context_tokens": 47280, + "processed_tokens": 680108, + "wall_ms": 109627, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 18, + "context_tokens": 58735, + "processed_tokens": 796950, + "wall_ms": 125905, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 39682, + "processed_tokens": 506861, + "wall_ms": 112096, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 22, + "context_tokens": 44561, + "processed_tokens": 690757, + "wall_ms": 102048, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 24, + "tool_calls": 31, + "context_tokens": 63565, + "processed_tokens": 1196208, + "wall_ms": 198639, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 14, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 32, + "context_tokens": 72885, + "processed_tokens": 1197019, + "wall_ms": 192600, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 51822, + "processed_tokens": 619091, + "wall_ms": 101641, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 26, + "context_tokens": 70594, + "processed_tokens": 1112608, + "wall_ms": 176011, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 31, + "context_tokens": 84573, + "processed_tokens": 1483018, + "wall_ms": 206298, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 42404, + "processed_tokens": 475669, + "wall_ms": 108542, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 35213, + "processed_tokens": 212332, + "wall_ms": 71652, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 52522, + "processed_tokens": 395462, + "wall_ms": 97474, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 68187, + "processed_tokens": 546712, + "wall_ms": 199037, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.9375, + "excluded": false, + "proof_reads": 17, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 15, + "context_tokens": 62075, + "processed_tokens": 408423, + "wall_ms": 117317, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 83594, + "processed_tokens": 630707, + "wall_ms": 169380, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 13, + "context_tokens": 72981, + "processed_tokens": 426361, + "wall_ms": 128845, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 17, + "context_tokens": 67119, + "processed_tokens": 431903, + "wall_ms": 121556, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "findcode", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 14, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 59952, + "processed_tokens": 453885, + "wall_ms": 119649, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 29322, + "processed_tokens": 451199, + "wall_ms": 94064, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 21651, + "processed_tokens": 316436, + "wall_ms": 64755, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 29183, + "processed_tokens": 318206, + "wall_ms": 65658, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 43551, + "processed_tokens": 653134, + "wall_ms": 113724, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 53487, + "processed_tokens": 847063, + "wall_ms": 150364, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 23, + "tool_calls": 35, + "context_tokens": 51351, + "processed_tokens": 967362, + "wall_ms": 140421, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 35110, + "processed_tokens": 337290, + "wall_ms": 72595, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 22, + "context_tokens": 52145, + "processed_tokens": 836750, + "wall_ms": 128745, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 30832, + "processed_tokens": 330193, + "wall_ms": 97992, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 32807, + "processed_tokens": 321484, + "wall_ms": 56305, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 23, + "context_tokens": 41040, + "processed_tokens": 642999, + "wall_ms": 123773, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 33144, + "processed_tokens": 584960, + "wall_ms": 92063, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 30, + "tool_calls": 29, + "context_tokens": 58867, + "processed_tokens": 1377878, + "wall_ms": 180809, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 27, + "context_tokens": 50142, + "processed_tokens": 773658, + "wall_ms": 150437, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: anything but a positive number is refused", + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 15, + "context_tokens": 50224, + "processed_tokens": 661075, + "wall_ms": 129697, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 62141, + "processed_tokens": 937840, + "wall_ms": 152687, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 22, + "context_tokens": 58478, + "processed_tokens": 864759, + "wall_ms": 106707, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 36748, + "processed_tokens": 348554, + "wall_ms": 79243, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 29870, + "processed_tokens": 144847, + "wall_ms": 72810, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 36198, + "processed_tokens": 280491, + "wall_ms": 61783, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 10, + "context_tokens": 53234, + "processed_tokens": 262811, + "wall_ms": 97882, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 9, + "context_tokens": 36932, + "processed_tokens": 239226, + "wall_ms": 72703, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 12, + "context_tokens": 54074, + "processed_tokens": 314867, + "wall_ms": 97506, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 11, + "context_tokens": 56025, + "processed_tokens": 336695, + "wall_ms": 118385, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 45068, + "processed_tokens": 184210, + "wall_ms": 94918, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "none", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.5714, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 11, + "context_tokens": 41785, + "processed_tokens": 220837, + "wall_ms": 82972, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 23, + "tool_calls": 22, + "context_tokens": 39310, + "processed_tokens": 848475, + "wall_ms": 78323, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 4, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 29957, + "processed_tokens": 375935, + "wall_ms": 78013, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 26175, + "processed_tokens": 295946, + "wall_ms": 64423, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 16, + "context_tokens": 35697, + "processed_tokens": 584582, + "wall_ms": 85573, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 32, + "context_tokens": 64911, + "processed_tokens": 971912, + "wall_ms": 141424, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 18, + "tool_calls": 20, + "context_tokens": 62613, + "processed_tokens": 903390, + "wall_ms": 130039, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 15, + "tool_calls": 21, + "context_tokens": 44071, + "processed_tokens": 593511, + "wall_ms": 90058, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 45020, + "processed_tokens": 770371, + "wall_ms": 111427, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 16, + "context_tokens": 35016, + "processed_tokens": 419615, + "wall_ms": 76879, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 19, + "context_tokens": 58250, + "processed_tokens": 882755, + "wall_ms": 113552, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 26, + "context_tokens": 50866, + "processed_tokens": 761051, + "wall_ms": 124981, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 16, + "tool_calls": 17, + "context_tokens": 50148, + "processed_tokens": 692628, + "wall_ms": 83249, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 53277, + "processed_tokens": 557661, + "wall_ms": 140663, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 57434, + "processed_tokens": 896685, + "wall_ms": 152971, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: Scale selection scales sizes and positions about the centre of the selection", + "acceptance: a percent sign is fine, and smaller works too", + "acceptance: a single layer scales about its own centre", + "acceptance: what is inside a selected group or frame scales with it", + "acceptance: it is one step to undo", + "acceptance: anything but a positive number is refused" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 21, + "context_tokens": 60563, + "processed_tokens": 937336, + "wall_ms": 134648, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 21, + "tool_calls": 24, + "context_tokens": 78094, + "processed_tokens": 1215052, + "wall_ms": 163938, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 24, + "context_tokens": 83894, + "processed_tokens": 1155234, + "wall_ms": 161395, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 40258, + "processed_tokens": 413448, + "wall_ms": 58574, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-K1", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 35857, + "processed_tokens": 211476, + "wall_ms": 67843, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-K2", + "kind": "plan", + "category": "control", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 8, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 50318, + "processed_tokens": 323556, + "wall_ms": 75949, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-P1", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 13, + "context_tokens": 67419, + "processed_tokens": 376719, + "wall_ms": 135916, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-P2", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 12, + "context_tokens": 67358, + "processed_tokens": 540368, + "wall_ms": 116282, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-P3", + "kind": "plan", + "category": "proof-only", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 82976, + "processed_tokens": 817838, + "wall_ms": 173885, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 78615, + "processed_tokens": 642865, + "wall_ms": 168485, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 9, + "tool_calls": 12, + "context_tokens": 62381, + "processed_tokens": 406937, + "wall_ms": 112533, + "model": "claude-haiku-5-5" + }, + { + "run": "r2", + "arm": "skill", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 8, + "tool_calls": 11, + "context_tokens": 53711, + "processed_tokens": 317363, + "wall_ms": 71786, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 46095, + "processed_tokens": 667486, + "wall_ms": 108330, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 36797, + "processed_tokens": 373826, + "wall_ms": 66961, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 33053, + "processed_tokens": 371257, + "wall_ms": 68068, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 12, + "context_tokens": 47252, + "processed_tokens": 475889, + "wall_ms": 72383, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 27, + "tool_calls": 38, + "context_tokens": 85128, + "processed_tokens": 1976061, + "wall_ms": 173239, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 20, + "tool_calls": 21, + "context_tokens": 62823, + "processed_tokens": 1080518, + "wall_ms": 114911, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 51253, + "processed_tokens": 592934, + "wall_ms": 89636, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 21, + "context_tokens": 62020, + "processed_tokens": 916160, + "wall_ms": 100484, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 53397, + "processed_tokens": 623797, + "wall_ms": 108341, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 58556, + "processed_tokens": 620543, + "wall_ms": 80737, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 20, + "context_tokens": 55148, + "processed_tokens": 719910, + "wall_ms": 94615, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 41112, + "processed_tokens": 396618, + "wall_ms": 75284, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 18, + "tool_calls": 19, + "context_tokens": 78283, + "processed_tokens": 1169702, + "wall_ms": 198104, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 39, + "context_tokens": 70530, + "processed_tokens": 1175257, + "wall_ms": 158643, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 24, + "tool_calls": 23, + "context_tokens": 80223, + "processed_tokens": 1696344, + "wall_ms": 152114, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 24, + "context_tokens": 85789, + "processed_tokens": 1310213, + "wall_ms": 168469, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 25, + "context_tokens": 76646, + "processed_tokens": 1248721, + "wall_ms": 149589, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "brief", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 22, + "context_tokens": 62385, + "processed_tokens": 800220, + "wall_ms": 95599, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 18, + "context_tokens": 59347, + "processed_tokens": 814804, + "wall_ms": 84437, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 10, + "tool_calls": 16, + "context_tokens": 54979, + "processed_tokens": 554231, + "wall_ms": 59992, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 58069, + "processed_tokens": 695021, + "wall_ms": 79878, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 15, + "context_tokens": 62676, + "processed_tokens": 613815, + "wall_ms": 81045, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 30, + "tool_calls": 39, + "context_tokens": 96051, + "processed_tokens": 2574600, + "wall_ms": 164536, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 70711, + "processed_tokens": 1471167, + "wall_ms": 117044, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 17, + "context_tokens": 62142, + "processed_tokens": 1003674, + "wall_ms": 95548, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 89840, + "processed_tokens": 1367524, + "wall_ms": 126178, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 72229, + "processed_tokens": 961101, + "wall_ms": 99010, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 85216, + "processed_tokens": 1293903, + "wall_ms": 130689, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: cd /tmp"], + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 21, + "context_tokens": 68499, + "processed_tokens": 1118154, + "wall_ms": 102373, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 70770, + "processed_tokens": 812009, + "wall_ms": 75662, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: cd /tmp"], + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 23, + "tool_calls": 22, + "context_tokens": 99525, + "processed_tokens": 1958594, + "wall_ms": 193355, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 20, + "context_tokens": 77340, + "processed_tokens": 1357818, + "wall_ms": 137896, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 14, + "tool_calls": 15, + "context_tokens": 72849, + "processed_tokens": 962674, + "wall_ms": 85708, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 22, + "context_tokens": 102552, + "processed_tokens": 1502987, + "wall_ms": 172555, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 21, + "context_tokens": 104243, + "processed_tokens": 1552914, + "wall_ms": 144446, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "dump", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 14, + "tool_calls": 19, + "context_tokens": 80380, + "processed_tokens": 1051624, + "wall_ms": 92676, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 18, + "context_tokens": 30179, + "processed_tokens": 486985, + "wall_ms": 108299, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 1, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 22328, + "processed_tokens": 254493, + "wall_ms": 61385, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 26035, + "processed_tokens": 284388, + "wall_ms": 64266, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 33371, + "processed_tokens": 433488, + "wall_ms": 83929, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 25, + "tool_calls": 34, + "context_tokens": 64465, + "processed_tokens": 1198656, + "wall_ms": 165175, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /tmp"], + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 27, + "tool_calls": 26, + "context_tokens": 51730, + "processed_tokens": 1109819, + "wall_ms": 157898, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 38616, + "processed_tokens": 365101, + "wall_ms": 96551, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 1, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 40480, + "processed_tokens": 429222, + "wall_ms": 87666, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 35641, + "processed_tokens": 392726, + "wall_ms": 87588, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 34913, + "processed_tokens": 479708, + "wall_ms": 91878, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 24, + "tool_calls": 23, + "context_tokens": 49266, + "processed_tokens": 939765, + "wall_ms": 118790, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 42505, + "processed_tokens": 574751, + "wall_ms": 96893, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 24, + "tool_calls": 28, + "context_tokens": 67978, + "processed_tokens": 1335467, + "wall_ms": 217634, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 24, + "context_tokens": 48834, + "processed_tokens": 924548, + "wall_ms": 129721, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 21, + "tool_calls": 21, + "context_tokens": 59685, + "processed_tokens": 982576, + "wall_ms": 136846, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 23, + "context_tokens": 66794, + "processed_tokens": 1032523, + "wall_ms": 169246, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 25, + "context_tokens": 70551, + "processed_tokens": 986081, + "wall_ms": 176490, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "find", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 40706, + "processed_tokens": 484023, + "wall_ms": 93592, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 35076, + "processed_tokens": 392160, + "wall_ms": 79036, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 14, + "context_tokens": 33491, + "processed_tokens": 424786, + "wall_ms": 92643, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 34973, + "processed_tokens": 294044, + "wall_ms": 84811, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 16, + "context_tokens": 39248, + "processed_tokens": 441550, + "wall_ms": 93371, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 17, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 28, + "tool_calls": 35, + "context_tokens": 66251, + "processed_tokens": 1347853, + "wall_ms": 217179, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 43966, + "processed_tokens": 489272, + "wall_ms": 116275, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 12, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 23, + "context_tokens": 53082, + "processed_tokens": 671906, + "wall_ms": 125161, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 16, + "tool_calls": 19, + "context_tokens": 58783, + "processed_tokens": 830873, + "wall_ms": 128280, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 15, + "tool_calls": 21, + "context_tokens": 56715, + "processed_tokens": 667359, + "wall_ms": 140516, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 21, + "context_tokens": 50225, + "processed_tokens": 686816, + "wall_ms": 112091, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /tmp"], + "proof_reads": 9, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 26, + "context_tokens": 51283, + "processed_tokens": 772155, + "wall_ms": 157937, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 15, + "context_tokens": 38774, + "processed_tokens": 498578, + "wall_ms": 100218, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": [ + "Read: /src/panel/schema.ts", + "Read: /src/panel/mixed.ts" + ], + "proof_reads": 14, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 47, + "context_tokens": 74820, + "processed_tokens": 1044503, + "wall_ms": 204641, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 19, + "context_tokens": 55726, + "processed_tokens": 724039, + "wall_ms": 151965, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: anything but a positive number is refused", + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 13, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 52812, + "processed_tokens": 939461, + "wall_ms": 138997, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 7, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 26, + "tool_calls": 34, + "context_tokens": 87910, + "processed_tokens": 1738373, + "wall_ms": 228381, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": true, + "left_folder_evidence": [ + "Bash: /p4/dbg-selsize.ts", + "Bash: /p4/dbg-selsize.ts", + "Bash: /p4/dbg-selsize.ts" + ], + "proof_reads": 14, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 25, + "tool_calls": 38, + "context_tokens": 86993, + "processed_tokens": 1728868, + "wall_ms": 223803, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "findcode", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 21, + "context_tokens": 49049, + "processed_tokens": 575393, + "wall_ms": 115130, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 25541, + "processed_tokens": 272995, + "wall_ms": 60117, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 18290, + "processed_tokens": 220329, + "wall_ms": 52815, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 30155, + "processed_tokens": 382057, + "wall_ms": 75834, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 18, + "context_tokens": 33297, + "processed_tokens": 420607, + "wall_ms": 95008, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": [ + "Bash: /p4/base", + "Bash: /p4/base/", + "Bash: /p4/base/*.ts", + "Bash: /p4/base" + ], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 31, + "context_tokens": 59356, + "processed_tokens": 848741, + "wall_ms": 122044, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": [ + "Bash: /p4/table.txt", + "Bash: /p4/table.txt", + "Bash: /p4/table.txt", + "Bash: cd /tmp" + ], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 26, + "tool_calls": 25, + "context_tokens": 47282, + "processed_tokens": 1009854, + "wall_ms": 129788, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 16, + "context_tokens": 37764, + "processed_tokens": 555544, + "wall_ms": 90824, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 43543, + "processed_tokens": 485129, + "wall_ms": 89896, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 29393, + "processed_tokens": 285881, + "wall_ms": 75225, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 12, + "context_tokens": 32540, + "processed_tokens": 385683, + "wall_ms": 66283, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 26, + "context_tokens": 36146, + "processed_tokens": 409371, + "wall_ms": 100016, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 31188, + "processed_tokens": 291436, + "wall_ms": 57386, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 29, + "context_tokens": 56606, + "processed_tokens": 781059, + "wall_ms": 167811, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 45531, + "processed_tokens": 769121, + "wall_ms": 115157, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 48093, + "processed_tokens": 500201, + "wall_ms": 98158, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": ["Bash: /tmp/claude-0"], + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 18, + "context_tokens": 50735, + "processed_tokens": 714652, + "wall_ms": 123818, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 22, + "context_tokens": 61283, + "processed_tokens": 857416, + "wall_ms": 133421, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "none", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 16, + "context_tokens": 40046, + "processed_tokens": 515535, + "wall_ms": 117349, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-K1", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 31635, + "processed_tokens": 399618, + "wall_ms": 106758, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-K2", + "kind": "build", + "category": "control", + "all_green": false, + "acceptance": true, + "constraints": true, + "regressions": false, + "failing": ["regressions: parseColor reads \\#rrggbb in any case"], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 22357, + "processed_tokens": 269877, + "wall_ms": 73564, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-K3", + "kind": "build", + "category": "control", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 34803, + "processed_tokens": 365852, + "wall_ms": 73249, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-P1", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 34365, + "processed_tokens": 373534, + "wall_ms": 67479, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-P3", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "left_folder_evidence": [ + "Bash: /p4/base-persist.test.ts", + "Bash: /p4/base-persist.test.ts" + ], + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 28, + "tool_calls": 40, + "context_tokens": 71058, + "processed_tokens": 1565906, + "wall_ms": 170647, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-P5", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 26, + "tool_calls": 27, + "context_tokens": 64958, + "processed_tokens": 1424614, + "wall_ms": 142915, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-P6", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 45248, + "processed_tokens": 638034, + "wall_ms": 114674, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-P7", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 20, + "tool_calls": 24, + "context_tokens": 63585, + "processed_tokens": 1022077, + "wall_ms": 128651, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-P8", + "kind": "build", + "category": "proof-only", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 38897, + "processed_tokens": 473735, + "wall_ms": 100712, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 14, + "tool_calls": 14, + "context_tokens": 39302, + "processed_tokens": 486959, + "wall_ms": 65217, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 21, + "tool_calls": 27, + "context_tokens": 53750, + "processed_tokens": 961980, + "wall_ms": 122540, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 42798, + "processed_tokens": 455867, + "wall_ms": 61583, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 3, + "timed_out": false, + "turns": 29, + "tool_calls": 29, + "context_tokens": 74750, + "processed_tokens": 1607384, + "wall_ms": 210790, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 25, + "tool_calls": 24, + "context_tokens": 60087, + "processed_tokens": 1205357, + "wall_ms": 168644, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 17, + "tool_calls": 18, + "context_tokens": 59672, + "processed_tokens": 828066, + "wall_ms": 133315, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 19, + "tool_calls": 22, + "context_tokens": 80182, + "processed_tokens": 1231333, + "wall_ms": 199699, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 1, + "timed_out": false, + "turns": 25, + "tool_calls": 28, + "context_tokens": 75197, + "processed_tokens": 1441220, + "wall_ms": 180798, + "model": "claude-haiku-5-5" + }, + { + "run": "r3", + "arm": "skill", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "proof_search_exposure": 2, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 59158, + "processed_tokens": 582151, + "wall_ms": 108012, + "model": "claude-haiku-5-5" + } +] diff --git a/benchmarks/proof-build/scripts/arms.mjs b/benchmarks/proof-build/scripts/arms.mjs new file mode 100644 index 00000000..6874f9b4 --- /dev/null +++ b/benchmarks/proof-build/scripts/arms.mjs @@ -0,0 +1,108 @@ +// The task text one build-benchmark session receives, for each arm. The +// part that is not about Proof is the same in every arm: the folder rule, +// the request, and the closing line. Arms differ only in the block between +// the folder rule and the request, and in what their clone holds (see +// CLONE). Nothing here names a task, a rule, or a record. + +export const ARMS = [ + 'none', + 'skill', + 'brief', + 'find', + 'findcode', + 'dump', + 'dump10x', +]; + +/** + * What each arm's clone is made from and holds. `bed` is the bed variant + * (`none`, `proof`, or `proof10x`); `proof` adds the skill, the + * `./flatbread` wrapper, the node_modules link, and the AGENTS.md line; + * `inject` names the text pasted into the task (`brief` or `dump`). + */ +export const CLONE = { + none: { bed: 'none', proof: false }, + skill: { bed: 'proof', proof: true }, + brief: { bed: 'proof', proof: true, inject: 'brief' }, + find: { bed: 'proof', proof: true }, + findcode: { bed: 'proof', proof: true }, + dump: { bed: 'proof', proof: true, inject: 'dump' }, + dump10x: { bed: 'proof10x', proof: true, inject: 'dump' }, +}; + +/** The line every Proof arm's AGENTS.md gains. `none` has no such line. */ +export const AGENTS_LINE = + 'Project memory (decisions, constraints, findings) lives in Proof; read it with `./flatbread proof …` (see the proof skill).'; + +/** Every arm's CLAUDE.md, so AGENTS.md loads the same way in each. */ +export const CLAUDE_MD = '@AGENTS.md\n'; + +const FIND = + 'Before planning, run `./flatbread proof find --all`, then `./flatbread proof get ` for any hit that bears on the request.'; +const FIND_CODE = + 'Before editing a file, run `./flatbread proof find --path `. Before relying on a Decision, run `./flatbread proof find --refs all`.'; + +/** The arm's block. `inject` is the brief or dump text for those arms. */ +function armBlock(arm, inject) { + switch (arm) { + case 'none': + case 'skill': + return ''; + case 'brief': + return `## Project memory briefs + +The output of \`./flatbread proof brief \` for each Effort in this repository. + +${inject} +`; + case 'find': + return `## Before you start + +${FIND} +`; + case 'findcode': + return `## Before you start + +${FIND} +${FIND_CODE} +`; + case 'dump': + case 'dump10x': + return `## Project memory records + +Every live record in this repository's project memory, grouped by Effort. + +${inject} +`; + default: + throw new Error(`Unknown arm ${arm}`); + } +} + +export const CLOSING = { + build: 'Implement it, add tests, and run the test suite.', + plan: 'Write PLAN.md in the repository root; do not change code.', +}; + +/** + * Markdown for one session: `kind` is `build` or `plan`, `prompt` the + * request in the user's words, `folder` the clone's absolute path, and + * `inject` the pasted text for `brief` and `dump` arms. + */ +export function armText({ arm, kind, prompt, folder, inject = '' }) { + if (!ARMS.includes(arm)) throw new Error(`Unknown arm ${arm}`); + if (!CLOSING[kind]) throw new Error(`Unknown task kind ${kind}`); + if (CLONE[arm].inject && !inject.trim()) + throw new Error(`Arm ${arm} needs its ${CLONE[arm].inject} text`); + const block = armBlock(arm, inject.trim()); + return `# Task + +You are working in a git repository at \`${folder}\`. Work only inside this folder: do not read, list, or search anything outside it. +${block ? `\n${block}` : ''} +## Request + +${prompt.trim()} + +${CLOSING[kind]} +`; +} diff --git a/benchmarks/proof-build/scripts/check-bed.mjs b/benchmarks/proof-build/scripts/check-bed.mjs new file mode 100644 index 00000000..93af17d0 --- /dev/null +++ b/benchmarks/proof-build/scripts/check-bed.mjs @@ -0,0 +1,263 @@ +#!/usr/bin/env node +// Checks built test beds against each other and against the design, for +// free, before any paid session: +// +// node scripts/check-bed.mjs --proof --none [--x10 ] +// +// - every commit has the same code tree in every variant (`git ls-tree` +// with .flatbread-proof/ and flatbread.config.js left out), and the same +// subject, author and date; +// - the none variant has no graph, no config and no `Proof:` trailer in any +// commit, and no rule wording (spec/banned.mjs BANNED_IN_CODE) anywhere in +// its history, code or messages; +// - the proof variants have trailers on the commits spec/proof.mjs says and +// on no others, and the same record ids for the real Efforts; +// - each drift commit carries no trailer and touches no graph file; +// - the x10 graph holds the real graph plus filler Efforts only. +// +// It prints one line per check and exits non-zero on the first failure. +import { execFileSync } from 'node:child_process'; +import { existsSync, readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; +import { COMMITS } from '../spec/history.mjs'; +import { PROOF, DRIFTS, EFFORTS } from '../spec/proof.mjs'; +import { BANNED_IN_CODE } from '../spec/banned.mjs'; +import { FILLER_KEYS } from '../spec/filler.mjs'; + +const PROOF_DIR = '.flatbread-proof'; +const CONFIG = 'flatbread.config.js'; + +function args(argv) { + const out = {}; + for (let i = 0; i < argv.length; i++) { + const flag = argv[i].replace(/^--/, ''); + if (!['proof', 'none', 'x10'].includes(flag)) + throw new Error(`Unknown argument ${argv[i]}`); + out[flag] = resolve(argv[++i]); + } + if (!out.proof || !out.none) + throw new Error( + 'usage: check-bed.mjs --proof --none [--x10 ]' + ); + return out; +} + +const git = (dir, argv) => + execFileSync('git', argv, { + cwd: dir, + encoding: 'utf8', + maxBuffer: 1 << 30, + env: { + PATH: process.env.PATH, + GIT_CONFIG_NOSYSTEM: '1', + GIT_CONFIG_GLOBAL: '/dev/null', + }, + }).trim(); + +let failed = false; +function check(label, ok, detail = '') { + console.log( + `${ok ? 'ok ' : 'FAIL'} ${label}${detail ? ` (${detail})` : ''}` + ); + if (!ok) failed = true; +} + +function log(dir) { + const shas = git(dir, ['rev-list', '--reverse', 'main']).split('\n'); + return shas.map((sha) => ({ + sha, + subject: git(dir, ['log', '-1', '--format=%s', sha]), + who: git(dir, ['log', '-1', '--format=%an <%ae> %aI | %cn <%ce> %cI', sha]), + trailers: git(dir, [ + 'log', + '-1', + '--format=%(trailers:key=Proof,valueonly,separator=%x2C)', + sha, + ]) + .split(',') + .map((s) => s.trim()) + .filter(Boolean), + code: git(dir, ['ls-tree', '-r', sha]) + .split('\n') + .filter( + (l) => !l.includes(`\t${PROOF_DIR}/`) && !l.endsWith(`\t${CONFIG}`) + ) + .join('\n'), + graph: git(dir, ['ls-tree', '-r', '--name-only', sha]) + .split('\n') + .filter((p) => p.startsWith(`${PROOF_DIR}/`)), + })); +} + +function keymap(dir) { + return JSON.parse(readFileSync(`${dir}.keymap.json`, 'utf8')); +} + +function main() { + const o = args(process.argv.slice(2)); + const beds = { + proof: o.proof, + none: o.none, + ...(o.x10 ? { x10: o.x10 } : {}), + }; + const logs = Object.fromEntries( + Object.entries(beds).map(([k, d]) => [k, log(d)]) + ); + const maps = Object.fromEntries( + Object.entries(beds).map(([k, d]) => [k, keymap(d)]) + ); + for (const [name, l] of Object.entries(logs)) + check( + `${name}: ${COMMITS.length} commits on main`, + l.length === COMMITS.length, + `${l.length}` + ); + + // Same code tree, subject, author and date, commit for commit. + for (const other of Object.keys(beds).filter((k) => k !== 'none')) { + const diff = COMMITS.map((c, i) => i).filter( + (i) => logs.none[i].code !== logs[other][i].code + ); + check( + `${other} and none have identical code trees at every commit`, + diff.length === 0, + diff.length + ? `differ at ${diff.map((i) => COMMITS[i].key)}` + : `${COMMITS.length} trees` + ); + const meta = COMMITS.map((c, i) => i).filter( + (i) => + logs.none[i].subject !== logs[other][i].subject || + logs.none[i].who !== logs[other][i].who + ); + check( + `${other} and none have the same subjects, authors and dates`, + meta.length === 0 + ); + } + check( + 'the tips of all variants have the same code tree', + new Set(Object.values(logs).map((l) => l.at(-1).code)).size === 1 + ); + + // The none variant has no Proof at all. + const none = logs.none; + check( + 'none: no Proof: trailer in any commit', + none.every((c) => !c.trailers.length) + ); + check( + 'none: no .flatbread-proof/ or flatbread.config.js in any commit', + none.every((c) => !c.graph.length) && + !git(o.none, ['log', '--all', '--format=%H', '--', CONFIG]) + ); + check('none: no record ids in its key map', !maps.none.records); + const history = git(o.none, [ + 'log', + '-p', + '--all', + '--format=%B', + ]).toLowerCase(); + const leaks = BANNED_IN_CODE.filter((p) => history.includes(p.toLowerCase())); + check( + 'none: no proof-only rule wording anywhere in its history (code, tests, docs, messages)', + leaks.length === 0, + leaks.join(', ') + ); + check( + 'none: no mention of Proof or flatbread in its history', + !/flatbread|proof:/i.test(history) + ); + + // Trailers where the spec puts them, and nowhere else. + for (const name of Object.keys(beds).filter((k) => k !== 'none')) { + const l = logs[name]; + const ids = maps[name].records; + const wrong = COMMITS.flatMap((c, i) => { + const want = (PROOF[c.key]?.trailers ?? []).map((k) => ids[k]).sort(); + const have = [...l[i].trailers].sort(); + return want.join() === have.join() ? [] : [c.key]; + }); + check( + `${name}: trailers match spec/proof.mjs`, + wrong.length === 0, + wrong.length + ? wrong.join(', ') + : `${l.filter((c) => c.trailers.length).length} commits with trailers` + ); + for (const [key, d] of Object.entries(DRIFTS)) { + const i = COMMITS.findIndex((c) => c.key === d.commit); + const touched = git(beds[name], [ + 'show', + '--name-only', + '--format=', + l[i].sha, + ]).split('\n'); + check( + `${name}: ${key} (${d.commit} "${COMMITS[i].subject}") has no trailer and leaves the graph alone`, + !l[i].trailers.length && + !touched.some((p) => p.startsWith(`${PROOF_DIR}/`)) + ); + } + const after = COMMITS.findIndex((c) => c.key === 'c76'); + const graphTouched = l.slice(after).filter((c, j) => { + const touched = git(beds[name], [ + 'show', + '--name-only', + '--format=', + c.sha, + ]).split('\n'); + return ( + touched.some((p) => p.startsWith(`${PROOF_DIR}/`)) && name === 'proof' + ); + }); + if (name === 'proof') + check( + 'proof: no commit from the first drift on touches the graph', + graphTouched.length === 0, + graphTouched.map((c) => c.subject).join('; ') + ); + } + + // The same record ids for the real Efforts in proof and x10. + if (maps.x10) { + const real = Object.keys(EFFORTS); + const p = maps.proof.records; + const x = maps.x10.records; + const differ = Object.keys(p).filter((k) => p[k] !== x[k]); + check( + 'x10 has every real record with the same id', + differ.length === 0, + differ.join(', ') + ); + const extra = Object.keys(x).filter((k) => !(k in p)); + check( + 'x10 adds only filler Efforts', + extra.every((k) => FILLER_KEYS.includes(k.split('.')[0])), + `${extra.length} filler keys` + ); + check( + 'x10 filler is outside the real Efforts', + extra.every((k) => !real.includes(k.split('.')[0])) + ); + const files = logs.x10.at(-1).graph.filter((f) => f.endsWith('.md')).length; + const proofFiles = logs.proof + .at(-1) + .graph.filter((f) => f.endsWith('.md')).length; + check( + 'x10 graph is about ten times the real one', + files >= 8 * proofFiles, + `${files} vs ${proofFiles} record files` + ); + } + for (const [name, dir] of Object.entries(beds)) + check( + `${name}: working tree clean, main checked out`, + !git(dir, ['status', '--porcelain']) && + git(dir, ['rev-parse', '--abbrev-ref', 'HEAD']) === 'main' && + existsSync(dir) + ); + if (failed) process.exitCode = 1; +} + +main(); diff --git a/benchmarks/proof-build/scripts/common.mjs b/benchmarks/proof-build/scripts/common.mjs new file mode 100644 index 00000000..664150ac --- /dev/null +++ b/benchmarks/proof-build/scripts/common.mjs @@ -0,0 +1,600 @@ +// What run.mjs, grade.mjs, judge.mjs and compare.mjs share: where a session +// keeps its files, how tasks load, git, and the transcript scans that flag +// leaks and count Proof use. +import { execFileSync, spawn } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { existsSync, readdirSync, readFileSync } from 'node:fs'; +import { readFile } from 'node:fs/promises'; +import { dirname, join, resolve, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +export const BENCH = resolve(dirname(fileURLToPath(import.meta.url)), '..'); +export const ROOT = resolve(BENCH, '..', '..'); +export const FLATBREAD_BIN = join( + ROOT, + 'packages', + 'flatbread', + 'bin', + 'flatbread.js' +); + +/** Files a session's clone gains from the runner, kept out of git status. */ +export const SESSION_FILES = [ + '/flatbread', + '/node_modules', + '/CLAUDE.md', + '/.claude/', + '/.flatbread/', +]; + +/** + * Paths left out of `session.diff`: Proof's own files, the runner's files, + * and PLAN.md, which is captured on its own. + */ +export const DIFF_EXCLUDE = [ + '.flatbread-proof', + '.flatbread', + '.claude', + 'AGENTS.md', + 'CLAUDE.md', + 'flatbread.config.js', + 'flatbread', + 'node_modules', + 'PLAN.md', +]; + +/** Files in a task folder that never reach a session or a grading tree. */ +const TASK_PRIVATE = /^(task\.json|rubric\.json|.*\.patch|plans)$/; + +/** A test file, as the bed's own suite names them. */ +export const TEST_FILE = /\.test\.(ts|mts|mjs|js)$/; + +export function git(cwd, ...argv) { + return execFileSync('git', argv, { + cwd, + encoding: 'utf8', + maxBuffer: 256 * 1024 * 1024, + stdio: ['ignore', 'pipe', 'pipe'], + env: { + ...process.env, + GIT_CONFIG_NOSYSTEM: '1', + GIT_CONFIG_GLOBAL: '/dev/null', + GIT_AUTHOR_NAME: 'bench', + GIT_AUTHOR_EMAIL: 'bench@example.invalid', + GIT_COMMITTER_NAME: 'bench', + GIT_COMMITTER_EMAIL: 'bench@example.invalid', + }, + }); +} + +export const sha256 = (text) => createHash('sha256').update(text).digest('hex'); + +/** tune or holdout, by the first byte of sha256(id), as in proof-recall. */ +export function splitOf(id) { + return createHash('sha256').update(id).digest()[0] % 2 === 0 + ? 'tune' + : 'holdout'; +} + +export const isUnder = (path, dir) => + path === dir || path.startsWith(dir.endsWith(sep) ? dir : dir + sep); + +/** Whether `path` lies in another session's clone beside `clone`. */ +export function isSiblingClone(path, clone) { + const parent = dirname(clone); + if (!isUnder(path, parent) || path === parent) return false; + const name = path.slice(parent.length + 1).split(sep)[0]; + const own = clone.slice(parent.length + 1); + return ( + name !== own && + name.length === own.length && + /^[0-9a-f]+$/.test(name) && + /^[0-9a-f]+$/.test(own) + ); +} + +/** Every task under `dir`: one folder per task holding task.json. */ +export function loadTasks(dir) { + const tasks = []; + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (!entry.isDirectory() || entry.name.startsWith('_')) continue; + const file = join(dir, entry.name, 'task.json'); + if (!existsSync(file)) continue; + const task = JSON.parse(readFileSync(file, 'utf8')); + if (task.id !== entry.name) + throw new Error(`${file}: id ${task.id} differs from its folder`); + if (!['build', 'plan'].includes(task.kind)) + throw new Error(`${file}: kind must be build or plan`); + if (!task.prompt?.trim()) throw new Error(`${file}: no prompt`); + tasks.push({ ...task, dir: join(dir, entry.name) }); + } + return tasks.sort((a, b) => a.id.localeCompare(b.id)); +} + +/** + * Tasks picked by `--only` (kinds or categories) and `--ids`. Both empty + * means every task. + */ +export function pickTasks(tasks, { only = [], ids = [] }) { + const known = new Set(tasks.map((t) => t.id)); + for (const id of ids) if (!known.has(id)) throw new Error(`No task ${id}`); + return tasks.filter( + (t) => + (!ids.length || ids.includes(t.id)) && + (!only.length || only.includes(t.kind) || only.includes(t.category)) + ); +} + +/** Hidden files a task copies into `test/_hidden/` at grading time. */ +export function hiddenFiles(taskDir) { + return readdirSync(taskDir).filter((name) => !TASK_PRIVATE.test(name)); +} + +/** Where one session keeps its files, outside its clone. */ +export function sessionDir(work, name, arm, id) { + return join(work, name, arm, id); +} + +/** The clone's path, which names neither the arm nor the task. */ +export function cloneDir(work, name, arm, id) { + return join( + work, + name, + 'clones', + sha256(`${name}\0${arm}\0${id}`).slice(0, 12) + ); +} + +export async function readJson(path, fallback = null) { + try { + return JSON.parse(await readFile(path, 'utf8')); + } catch { + return fallback; + } +} + +/** Every session folder under `/`: [{arm, id, dir}]. */ +export function listSessions(work, name) { + const base = join(work, name); + const out = []; + if (!existsSync(base)) return out; + for (const arm of readdirSync(base, { withFileTypes: true })) { + if (!arm.isDirectory() || arm.name === 'clones') continue; + for (const id of readdirSync(join(base, arm.name), { + withFileTypes: true, + })) + if (id.isDirectory()) + out.push({ + arm: arm.name, + id: id.name, + dir: join(base, arm.name, id.name), + }); + } + return out; +} + +/** + * Runs a command with a hard timeout, killing its whole process group. + * Resolves with { code, timedOut, stdout, stderr, ms }. + */ +export function runCommand(cmd, argv, { cwd, env, timeout, input } = {}) { + return new Promise((done) => { + const start = Date.now(); + const child = spawn(cmd, argv, { + cwd, + env, + detached: true, + stdio: [input === undefined ? 'ignore' : 'pipe', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + let timedOut = false; + child.stdout.on('data', (c) => (stdout += c)); + child.stderr.on('data', (c) => (stderr += c)); + if (input !== undefined) child.stdin.end(input); + const kill = (signal) => { + try { + process.kill(-child.pid, signal); + } catch { + // Already gone. + } + }; + const timer = + timeout && + setTimeout(() => { + timedOut = true; + kill('SIGTERM'); + setTimeout(() => kill('SIGKILL'), 5000).unref(); + }, timeout); + child.on('close', (code) => { + clearTimeout(timer); + done({ code, timedOut, stdout, stderr, ms: Date.now() - start }); + }); + }); +} + +/** + * Tool calls in a stream-json transcript: [{name, input, result}], where + * `result` is the text of the matching tool_result, when there is one. + */ +export async function toolCalls(transcriptPath) { + let text = ''; + try { + text = await readFile(transcriptPath, 'utf8'); + } catch { + return []; + } + const calls = []; + const byId = new Map(); + for (const line of text.split('\n')) { + let entry; + try { + entry = JSON.parse(line); + } catch { + continue; + } + for (const part of entry.message?.content ?? []) { + if ( + entry.type === 'assistant' && + part.type === 'tool_use' && + !byId.has(part.id) + ) { + const call = { name: part.name, input: part.input ?? {} }; + byId.set(part.id, call); + calls.push(call); + } + if (entry.type === 'user' && part.type === 'tool_result') { + const call = byId.get(part.tool_use_id); + if (!call) continue; + const content = Array.isArray(part.content) + ? part.content + .map((c) => (typeof c === 'string' ? c : c.text ?? '')) + .join('\n') + : String(part.content ?? ''); + call.result = (call.result ?? '') + content; + } + } + } + return calls; +} + +/** The `init` and `result` events of a transcript. */ +export async function transcriptEnds(transcriptPath) { + const out = { init: null, result: null }; + let text = ''; + try { + text = await readFile(transcriptPath, 'utf8'); + } catch { + return out; + } + for (const line of text.split('\n')) { + let entry; + try { + entry = JSON.parse(line); + } catch { + continue; + } + if (entry.type === 'system' && entry.subtype === 'init') out.init = entry; + if (entry.type === 'result') out.result = entry; + } + return out; +} + +const READ_COMMANDS = new Set([ + 'list', + 'find', + 'get', + 'records', + 'brief', + 'relations', + 'blocking-decisions', + 'state', + 'help', +]); + +// Absolute paths in shell text: a slash after a start, space, quote, `=`, +// `(` or `:`, then at least one path character, so a lone `/` (division) +// and `//` or `/*` (comments) are not paths. +const ABS_PATH = /(?:^|[\s'"=(:<>])(\/[\w.@+%~,-]+(?:\/[\w.@+%~,*-]*)*)/g; +// Relative paths that start with `..`. +const DOTDOT = + /(?:^|[\s'"=(:<>])(\.\.(?:\/[\w.@+%~,*-]*)*)(?=$|[\s'"):;|&<>])/g; +// Commands whose bare `/` argument means the filesystem root. +const FS_COMMANDS = + /^\s*(?:sudo\s+)?(ls|find|du|tree|grep|rg|ag|cat|stat|locate|fd|head|tail|less|file|cp|mv|rm|tar)\b/; +// The script text of `node -e`, `python -c`, `sh -c` and the like. +const INLINE_SCRIPT = + /(\s(?:-e|-c|--eval|--print)\s+)('(?:[^']*)'|"(?:\\.|[^"\\])*")/g; + +/** + * Splits a Bash command into shell text and body text. Body text is what a + * program reads rather than what the shell resolves: heredoc bodies and + * inline scripts (`node -e "…"`). Only shell text is checked for `..` and + * relative paths; body text is still checked for absolute paths under the + * protected roots, since a heredoc can feed a shell. + */ +export function splitShell(command) { + const shell = []; + const body = []; + const lines = command.split('\n'); + for (let i = 0; i < lines.length; i++) { + let line = lines[i]; + line = line.replace(INLINE_SCRIPT, (_, flag, script) => { + body.push(script); + return `${flag}''`; + }); + shell.push(line); + const heredoc = line.match(/<<-?\s*(['"]?)([\w.-]+)\1/); + if (!heredoc) continue; + const end = heredoc[2]; + const strip = line.includes('<<-'); + for (i++; i < lines.length; i++) { + const text = strip ? lines[i].replace(/^\t+/, '') : lines[i]; + if (text === end) break; + body.push(lines[i]); + } + } + return { shell: shell.join('\n'), body: body.join('\n') }; +} + +const PROOF_CLI = /flatbread\s+proof\b/; + +/** + * Scans a session's tool calls for leaks and Proof use. + * + * - `left_folder`: a file tool's path, a Glob or Grep path or absolute + * pattern, or a path in a Bash command lies outside `clone`. In Bash, a + * relative path starts at the shell's folder, which follows `cd` inside + * the clone; a `cd` out of it counts as leaving. A relative path outside + * the clone counts only when it names one of the clone's parent folders, + * another session's clone, or something that exists under a protected + * root, since it may be a search pattern or a miscounted `../`. Absolute paths outside the + * clone count only when they are the clone's ancestors or fall under a + * `protected` root (the work, task, grade, hidden and bed folders, the + * p4 folder and this repository), so `/dev/null` or `/usr/bin/env` do + * not. Heredoc bodies and inline scripts are checked only for absolute + * paths under a protected root, so code such as `import '../src/x.ts'`, + * `100 / 3` or a `// comment` written through the shell is not a leak. + * - `hidden_read`: a tool input names a hidden root (the task, hidden and + * grading folders, the gate log), `test/_hidden`, or a task's reference or + * naive patch or rubric. + * - `proof_reads` / `proof_writes`: `flatbread proof ` calls, + * split by whether the command only reads; direct edits under + * `.flatbread-proof/` count as writes. + * - `proof_files_read`: Read calls on files under `.flatbread-proof/`. + * - `proof_search_exposure`: searches and other calls that are not Proof + * commands but whose result showed text from `.flatbread-proof/` (a + * repository-wide Grep, `grep -r`, `find`, `ls -R`). + * - `proof_touch`: any mention of `.flatbread-proof` or `flatbread proof`, + * which a `none` session should never make. + * - `opened_cited`: Read calls on a path a code Citation names. + */ +export function scanCalls( + calls, + { clone, protectedRoots = [], hiddenRoots = [], citedPaths = [] } +) { + const out = { + left_folder: false, + left_folder_evidence: [], + hidden_read: false, + hidden_read_evidence: [], + proof_reads: 0, + proof_writes: 0, + proof_files_read: 0, + proof_search_exposure: 0, + proof_touch: false, + skill_calls: 0, + opened_cited: 0, + }; + const cited = new Set(citedPaths); + const isProtected = (abs) => + isUnder(clone, abs) || protectedRoots.some((root) => isUnder(abs, root)); + // `cwd` is where a relative path starts: the clone for file tools, and + // the shell's folder for Bash, which follows `cd` within and across calls + // (as Claude Code's Bash keeps its folder while it stays in the project). + const outside = (path, fromShell, cwd = clone) => { + const abs = resolve(cwd, path); + if (isUnder(abs, clone)) return false; + if (!fromShell) return true; + if (path.startsWith('/')) return isProtected(abs); + // A relative path in a shell command may be a pattern or a string + // (`grep "from '../src/x"`) or a miscounted `../`, so it counts only when + // it reaches the clone's folders, another session's clone, or something + // that exists under a protected root. + return ( + isUnder(clone, abs) || + isSiblingClone(abs, clone) || + (protectedRoots.some((root) => isUnder(abs, root)) && existsSync(abs)) + ); + }; + let shellCwd = clone; + const flag = (path, call) => { + out.left_folder = true; + if (out.left_folder_evidence.length < 5) + out.left_folder_evidence.push(`${call.name}: ${path}`); + }; + for (const call of calls) { + const input = call.input ?? {}; + const text = JSON.stringify(input); + if (call.name === 'Skill') out.skill_calls++; + if ( + hiddenRoots.some((root) => text.includes(root)) || + /test\/_hidden|(reference|naive)[\w-]*\.patch|rubric\.json|gate\.(log|json)/.test( + text + ) + ) { + out.hidden_read = true; + if (out.hidden_read_evidence.length < 5) + out.hidden_read_evidence.push(`${call.name}: ${text.slice(0, 160)}`); + } + if (/\.flatbread-proof|flatbread\s+proof/.test(text)) + out.proof_touch = true; + const isProofCommand = + call.name === 'Bash' && PROOF_CLI.test(input.command ?? ''); + const readsProofFile = + ['Read', 'Write', 'Edit', 'NotebookEdit'].includes(call.name) && + /\.flatbread-proof\//.test(input.file_path ?? ''); + if ( + !isProofCommand && + !readsProofFile && + typeof call.result === 'string' && + /\.flatbread-proof\//.test(call.result) + ) + out.proof_search_exposure++; + const paths = [input.file_path, input.path, input.notebook_path].filter( + (p) => typeof p === 'string' && p + ); + if ( + ['Glob', 'Grep'].includes(call.name) && + typeof input.pattern === 'string' && + input.pattern.startsWith('/') + ) + paths.push(input.pattern); + for (const path of paths) if (outside(path, false)) flag(path, call); + if (call.name === 'Read' && typeof input.file_path === 'string') { + const rel = resolve(clone, input.file_path).slice(clone.length + 1); + if (cited.has(rel)) out.opened_cited++; + if (readsProofFile) out.proof_files_read++; + } + if ( + ['Write', 'Edit', 'NotebookEdit'].includes(call.name) && + /\.flatbread-proof\//.test(input.file_path ?? '') + ) + out.proof_writes++; + if (call.name === 'Bash' && typeof input.command === 'string') { + const command = input.command; + const { shell, body } = splitShell(command); + let cwd = shellCwd; + for (const segment of shell.split(/&&|\|\||;|\||\n/)) { + const cd = segment.match(/^\s*cd(?:\s+(\S+))?\s*$/); + if (cd) { + const arg = (cd[1] ?? '~').replace(/^['"]|['"]$/g, ''); + if (arg === '-') continue; + const target = /^[~$]/.test(arg) ? '/' : resolve(cwd, arg); + if (isUnder(target, clone)) cwd = target; + else { + flag(`cd ${arg}`, call); + cwd = clone; + } + continue; + } + if (FS_COMMANDS.test(segment) && /\s\/(?=\s|$)/.test(segment)) + flag('/', call); + for (const m of segment.matchAll(ABS_PATH)) + if (outside(m[1], true, cwd)) flag(m[1], call); + for (const m of segment.matchAll(DOTDOT)) + if (outside(m[1], true, cwd)) flag(m[1], call); + } + for (const m of body.matchAll(ABS_PATH)) { + const abs = resolve(m[1]); + if (!isUnder(abs, clone) && protectedRoots.some((r) => isUnder(abs, r))) + flag(m[1], call); + } + shellCwd = cwd; + for (const m of command.matchAll(/flatbread\s+proof\s+([a-z][\w-]*)/g)) + if (READ_COMMANDS.has(m[1])) out.proof_reads++; + else out.proof_writes++; + } + } + return out; +} + +/** Code paths that the graph's code Citations name (`path#symbol@sha`). */ +export function citedPaths(proofRoot) { + const dir = join(proofRoot, 'citations'); + if (!existsSync(dir)) return []; + const paths = new Set(); + for (const file of readdirSync(dir).filter((f) => f.endsWith('.md'))) { + const text = readFileSync(join(dir, file), 'utf8'); + const body = text.replace(/^---\n[\s\S]*?\n---\n/, ''); + for (const m of body.matchAll(/^([\w./-]+\.\w+)#[\w$.]+(?:@\w+)?/gm)) + paths.add(m[1]); + } + return [...paths]; +} + +/** Environment variables that tie a process to the session that started it. */ +export const SESSION_VARS = [ + 'CLAUDE_CODE_SESSION_ID', + 'CLAUDE_CODE_REMOTE_SESSION_ID', + 'CLAUDE_CODE_REMOTE_SDK_URL', + 'CLAUDE_CODE_MESSAGING_SOCKET', + 'CLAUDE_CODE_MESSAGING_TOKEN', + 'CLAUDE_SESSION_INGRESS_TOKEN_FILE', + 'CLAUDE_CODE_CHILD_SESSION', + 'CLAUDE_CODE_SESSION_ATTENDED', + 'CLAUDE_CODE_SYNC_SESSION_REFS', + 'CLAUDE_CODE_TEE_SDK_STDOUT', + 'CLAUDE_PID', + 'CLAUDECODE', + 'CLAUDE_CODE_ENTRYPOINT', + 'FLATBREAD_PROOF_SERVER', +]; + +export function cleanEnv() { + const env = { ...process.env }; + for (const key of SESSION_VARS) delete env[key]; + return env; +} + +/** Parses `--flag value` and bare arguments against a spec of defaults. */ +export function parseArgs(argv, spec, { lists = [], flags = [] } = {}) { + const out = { ...spec, _: [] }; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (!arg.startsWith('--')) { + out._.push(arg); + continue; + } + const key = arg.slice(2).replace(/-(\w)/g, (_, c) => c.toUpperCase()); + if (flags.includes(key)) { + out[key] = true; + continue; + } + if (!(key in spec)) throw new Error(`Unknown argument ${arg}`); + const value = argv[++i]; + if (value === undefined) throw new Error(`${arg} needs a value`); + out[key] = lists.includes(key) + ? value.split(',').filter(Boolean) + : typeof spec[key] === 'number' + ? Number(value) + : value; + } + return out; +} + +let netnsWorks = null; +/** + * `[cmd, argv]` that runs `cmd argv` with no network when this machine + * allows it (`unshare -n`, which needs root or user namespaces), and as + * it is otherwise. `isolated` says which. + */ +export function withoutNetwork(cmd, argv) { + if (netnsWorks === null) { + try { + execFileSync('unshare', ['-n', 'true'], { stdio: 'ignore' }); + netnsWorks = true; + } catch { + try { + execFileSync('unshare', ['-rn', 'true'], { stdio: 'ignore' }); + netnsWorks = '-rn'; + } catch { + netnsWorks = false; + } + } + } + if (!netnsWorks) return { cmd, argv, isolated: false }; + return { + cmd: 'unshare', + argv: [netnsWorks === '-rn' ? '-rn' : '-n', '--', cmd, ...argv], + isolated: true, + }; +} + +/** The environment for grading runs: no session variables, no proxy. */ +export function gradeEnv() { + const env = cleanEnv(); + for (const key of Object.keys(env)) + if (/^(https?|all|no)_proxy$/i.test(key)) delete env[key]; + return env; +} diff --git a/benchmarks/proof-build/scripts/compare.mjs b/benchmarks/proof-build/scripts/compare.mjs new file mode 100644 index 00000000..fc119269 --- /dev/null +++ b/benchmarks/proof-build/scripts/compare.mjs @@ -0,0 +1,459 @@ +#!/usr/bin/env node +// Compares every arm with `none`, paired by task and run. +// +// node scripts/compare.mjs --runs r1,r2 +// [--arms skill,brief,find,findcode,dump,dump10x] [--base none] +// [--split tune|holdout|all] [--only build|plan|] +// [--ids a,b] [--keep-flagged] [--errors drop|fail] [--out ] +// +// It reads each session's session.json, grade.json and judge.json under +// ////. A row is one (task, run) pair that both +// the arm and the base have graded. +// +// Exclusion: when any compared arm's session for a (task, run) is +// `excluded` by grade.mjs (left_folder, hidden_read, or a `none` session +// that found Proof), that (task, run) is dropped from every comparison, as +// DESIGN.md §7 says. --keep-flagged keeps them. Flag counts by arm are in +// `flags`. Timed-out sessions stay in and count as failures. Sessions that +// ended in a non-transient API error (run.mjs `api_error`) are counted in +// `flags` and, by default, their (task, run) is dropped like a flagged one; +// --errors fail keeps them as failures instead. +// +// Build tasks: all-green, acceptance, constraints and regressions, each +// with the arm's and the base's pass counts, the pairs only one side passed +// (`wins`, `losses`), a two-sided exact sign test over those pooled pairs +// (`p`), and the stricter task-level sign test that counts each task once, +// as a win for the side that passed more often over the runs +// (`taskSignTest`). A constraint pass counts only when acceptance passed +// too, because constraint tests skip when the feature is missing and an +// empty diff would otherwise pass them; `constraints_raw` is the plain +// count, for reference. +// +// Primary endpoints (DESIGN.md §7): all-green on proof-only tasks (H1) +// and on stale tasks (H3), each arm against `none`, ten sign tests +// Holm-corrected together (`holm_primary`, keyed `:`). The +// pooled all-green over every build task is secondary, Holm-corrected +// across the five primary arms (`holm_all_green`). `dump10x` is compared +// with `dump` too, without correction. +// +// Plan tasks: the mean rubric score of each side, a paired two-sided +// Wilcoxon signed-rank test (exact, ties averaged), and must-avoid +// violations. +// +// Cost, over build and plan pairs apart: first-turn tokens (the system +// prompt plus the task text, so the pasted brief or dump shows here), +// context tokens (every fresh input and output token, first turn +// included), read tokens (context tokens without the first turn: what the +// session read and wrote by itself), processed tokens (context plus cache +// reads, which counts a pasted block once per turn), tool calls, turns, +// wall time, Proof reads, direct reads of Proof files, and searches whose +// results showed Proof files (`proof_search_exposure`), as medians, with +// the median paired ratio arm/base for the token measures and wall time. +// H4 is read on context and processed tokens, which include what was +// forced in; read tokens alone would hide it. +// +// Everything is also given per category (`byCategory`), which is +// descriptive. --split keeps only the tune or holdout tasks, split by the +// first byte of sha256(task id) as in proof-recall. +import { writeFile } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { signTest } from '../../proof-recall/scripts/compare.mjs'; +import { ARMS } from './arms.mjs'; +import { listSessions, parseArgs, readJson, splitOf } from './common.mjs'; + +const PRIMARY = ['skill', 'brief', 'find', 'findcode', 'dump']; +const PRIMARY_CATEGORIES = ['proof-only', 'stale']; +const BUILD_GROUPS = [ + 'all_green', + 'acceptance', + 'constraints', + 'constraints_raw', + 'regressions', +]; +const RATIOS = [ + 'first_turn_tokens', + 'context_tokens', + 'read_tokens', + 'processed_tokens', + 'wall_ms', +]; + +function median(values) { + const v = values.filter((x) => Number.isFinite(x)).sort((a, b) => a - b); + if (!v.length) return null; + const mid = v.length / 2; + return v.length % 2 ? v[Math.floor(mid)] : (v[mid - 1] + v[mid]) / 2; +} + +const round = (x, digits = 0) => + x === null || x === undefined ? null : +x.toFixed(digits); + +/** Two-sided Wilcoxon signed-rank p for paired differences `d`. */ +export function wilcoxon(d) { + const nz = d.filter((x) => x !== 0); + const n = nz.length; + if (!n) return { n: 0, w_plus: 0, p: 1 }; + const order = nz + .map((x, i) => ({ abs: Math.abs(x), i })) + .sort((a, b) => a.abs - b.abs); + const ranks = new Array(n); + for (let i = 0; i < n; ) { + let j = i; + while (j + 1 < n && order[j + 1].abs === order[i].abs) j++; + for (let k = i; k <= j; k++) ranks[order[k].i] = (i + j + 2) / 2; + i = j + 1; + } + const wPlus = nz.reduce((s, x, i) => s + (x > 0 ? ranks[i] : 0), 0); + if (n <= 25) { + // Exact: the null distribution of W+ over all 2^n sign patterns, on + // doubled ranks so that tied (half) ranks stay integers. + const r2 = ranks.map((r) => Math.round(r * 2)); + const max = r2.reduce((a, b) => a + b, 0); + let dist = new Float64Array(max + 1); + dist[0] = 1; + for (const r of r2) { + const next = new Float64Array(max + 1); + for (let s = 0; s <= max; s++) + if (dist[s]) { + next[s] += dist[s] / 2; + next[s + r] += dist[s] / 2; + } + dist = next; + } + const w = Math.round(wPlus * 2); + let low = 0; + let high = 0; + for (let s = 0; s <= max; s++) { + if (s <= w) low += dist[s]; + if (s >= w) high += dist[s]; + } + return { n, w_plus: wPlus, p: Math.min(1, 2 * Math.min(low, high)) }; + } + const mean = (n * (n + 1)) / 4; + const ties = {}; + for (const r of ranks) ties[r] = (ties[r] ?? 0) + 1; + const tie = Object.values(ties).reduce((s, t) => s + (t ** 3 - t), 0); + const sd = Math.sqrt((n * (n + 1) * (2 * n + 1)) / 24 - tie / 48); + const z = (Math.abs(wPlus - mean) - 0.5) / sd; + // Two-sided normal tail, by the Abramowitz-Stegun erfc approximation. + const t = 1 / (1 + 0.3275911 * (z / Math.SQRT2)); + const erfc = + t * + (0.254829592 + + t * + (-0.284496736 + + t * (1.421413741 + t * (-1.453152027 + t * 1.061405429)))) * + Math.exp(-((z / Math.SQRT2) ** 2)); + return { n, w_plus: wPlus, p: Math.min(1, erfc) }; +} + +/** Holm step-down adjustment of `ps` (a map name → p). */ +export function holm(ps) { + const entries = Object.entries(ps).sort((a, b) => a[1] - b[1]); + const out = {}; + let running = 0; + entries.forEach(([name, p], i) => { + running = Math.max(running, Math.min(1, p * (entries.length - i))); + out[name] = +running.toFixed(4); + }); + return out; +} + +function options(argv) { + const o = parseArgs( + argv, + { + runs: [], + arms: [], + base: 'none', + split: 'all', + only: [], + ids: [], + out: null, + errors: 'drop', + }, + { lists: ['runs', 'arms', 'only', 'ids'], flags: ['keepFlagged'] } + ); + if (!['drop', 'fail'].includes(o.errors)) + throw new Error('--errors must be drop or fail'); + if (!o._[0] || !o.runs.length) + throw new Error('usage: compare.mjs --runs r1[,r2] [options]'); + if (!['tune', 'holdout', 'all'].includes(o.split)) + throw new Error('--split must be one of tune, holdout, all'); + o.dir = resolve(o._[0]); + return o; +} + +async function loadRows(o) { + const rows = []; + for (const run of o.runs) + for (const s of listSessions(o.dir, run)) { + const session = await readJson(join(s.dir, 'session.json')); + const grade = await readJson(join(s.dir, 'grade.json')); + if (!session?.done || !grade) continue; + const judge = await readJson(join(s.dir, 'judge.json')); + rows.push({ + key: `${s.id}#${run}`, + task: s.id, + run, + arm: s.arm, + kind: grade.kind, + category: grade.category ?? session.category ?? 'none', + session, + grade, + judge, + }); + } + return rows.filter( + (r) => + (o.split === 'all' || splitOf(r.task) === o.split) && + (!o.ids.length || o.ids.includes(r.task)) && + (!o.only.length || o.only.includes(r.kind) || o.only.includes(r.category)) + ); +} + +const readTokens = (r) => + (r.session.cost?.context_tokens ?? NaN) - (r.session.cost?.fixed_tokens ?? 0); +const metric = { + first_turn_tokens: (r) => r.session.cost?.fixed_tokens ?? NaN, + context_tokens: (r) => r.session.cost?.context_tokens ?? NaN, + read_tokens: readTokens, + processed_tokens: (r) => r.session.cost?.processed_tokens ?? NaN, + tool_calls: (r) => r.session.cost?.tool_calls ?? NaN, + turns: (r) => r.session.cost?.turns ?? NaN, + wall_ms: (r) => r.session.wall_ms ?? NaN, + proof_reads: (r) => r.grade.flags?.proof_reads ?? 0, + proof_files_read: (r) => r.grade.flags?.proof_files_read ?? 0, + proof_search_exposure: (r) => r.grade.flags?.proof_search_exposure ?? 0, +}; +const passed = (r, group) => { + if (group === 'all_green') return r.grade.all_green === true; + if (group === 'constraints') + return ( + r.grade.acceptance?.pass === true && r.grade.constraints?.pass !== false + ); + if (group === 'constraints_raw') return r.grade.constraints?.pass !== false; + return r.grade[group]?.pass !== false; +}; + +function binary(pairs, group) { + const wins = pairs.filter(([a, b]) => passed(a, group) && !passed(b, group)); + const losses = pairs.filter( + ([a, b]) => !passed(a, group) && passed(b, group) + ); + const byTask = new Map(); + for (const [a, b] of pairs) + byTask.set( + a.task, + (byTask.get(a.task) ?? 0) + + Number(passed(a, group)) - + Number(passed(b, group)) + ); + const tw = [...byTask.values()].filter((m) => m > 0).length; + const tl = [...byTask.values()].filter((m) => m < 0).length; + return { + arm: pairs.filter(([a]) => passed(a, group)).length, + base: pairs.filter(([, b]) => passed(b, group)).length, + wins: wins.map(([a]) => a.key), + losses: losses.map(([a]) => a.key), + p: +signTest(wins.length, losses.length).toFixed(4), + taskSignTest: { arm: tw, base: tl, p: +signTest(tw, tl).toFixed(4) }, + }; +} + +function plans(pairs) { + const scored = pairs.filter(([a, b]) => a.judge && b.judge); + const d = scored.map(([a, b]) => a.judge.score - b.judge.score); + const mean = (xs) => + xs.length ? +(xs.reduce((s, x) => s + x, 0) / xs.length).toFixed(3) : null; + return { + pairs: scored.length, + unjudged: pairs.length - scored.length, + score: { + arm: mean(scored.map(([a]) => a.judge.score)), + base: mean(scored.map(([, b]) => b.judge.score)), + }, + wins: d.filter((x) => x > 0).length, + losses: d.filter((x) => x < 0).length, + wilcoxon: (({ n, w_plus, p }) => ({ n, w_plus, p: +p.toFixed(4) }))( + wilcoxon(d) + ), + avoid_violations: { + arm: scored.reduce((n, [a]) => n + a.judge.avoid_violations, 0), + base: scored.reduce((n, [, b]) => n + b.judge.avoid_violations, 0), + }, + plan_changed_code: { + arm: pairs.filter(([a]) => a.grade.plan_changed_code).length, + base: pairs.filter(([, b]) => b.grade.plan_changed_code).length, + }, + }; +} + +function cost(pairs) { + const out = {}; + for (const [name, get] of Object.entries(metric)) { + const a = pairs.map(([x]) => get(x)); + const b = pairs.map(([, y]) => get(y)); + out[name] = { arm: round(median(a)), base: round(median(b)) }; + if (RATIOS.includes(name)) { + out[name].ratio = round( + median(pairs.map(([x, y]) => get(x) / get(y)).filter(Number.isFinite)), + 3 + ); + out[name].armCheaper = pairs.filter(([x, y]) => get(x) < get(y)).length; + } + } + return out; +} + +function compareArm(rows, arm, base) { + const index = new Map( + rows.filter((r) => r.arm === base).map((r) => [r.key, r]) + ); + const pairs = rows + .filter((r) => r.arm === arm && index.has(r.key)) + .map((r) => [r, index.get(r.key)]); + const build = pairs.filter(([a]) => a.kind === 'build'); + const plan = pairs.filter(([a]) => a.kind === 'plan'); + const section = (ps) => ({ + pairs: ps.length, + ...Object.fromEntries(BUILD_GROUPS.map((g) => [g, binary(ps, g)])), + }); + const categories = [...new Set(pairs.map(([a]) => a.category))].sort(); + return { + arm, + base, + build: section(build), + plan: plans(plan), + cost: { build: cost(build), plan: cost(plan) }, + byCategory: Object.fromEntries( + categories.map((c) => { + const b = build.filter(([a]) => a.category === c); + const p = plan.filter(([a]) => a.category === c); + return [ + c, + { + build: b.length + ? { + pairs: b.length, + all_green: binary(b, 'all_green'), + acceptance: binary(b, 'acceptance'), + constraints: binary(b, 'constraints'), + constraints_raw: binary(b, 'constraints_raw'), + } + : null, + plan: p.length ? plans(p) : null, + cost: (({ + first_turn_tokens, + context_tokens, + read_tokens, + processed_tokens, + wall_ms, + }) => ({ + first_turn_tokens, + context_tokens, + read_tokens, + processed_tokens, + wall_ms, + }))(cost([...b, ...p])), + }, + ]; + }) + ), + }; +} + +async function main() { + const o = options(process.argv.slice(2)); + let rows = await loadRows(o); + const present = [...new Set(rows.map((r) => r.arm))]; + const arms = ( + o.arms.length ? o.arms : ARMS.filter((a) => present.includes(a)) + ).filter((a) => a !== o.base); + const compared = new Set([o.base, ...arms]); + rows = rows.filter((r) => compared.has(r.arm)); + const flags = {}; + for (const r of rows) { + const f = (flags[r.arm] ??= { + sessions: 0, + excluded: 0, + left_folder: 0, + hidden_read: 0, + none_found_proof: 0, + timed_out: 0, + api_error: 0, + apply_failed: 0, + edited_existing_tests: 0, + edited_test_support: 0, + }); + f.sessions++; + for (const k of [ + 'excluded', + 'left_folder', + 'hidden_read', + 'none_found_proof', + 'timed_out', + ]) + if (r.grade.flags?.[k]) f[k]++; + if (r.session.api_error) f.api_error++; + if (r.grade.apply_failed) f.apply_failed++; + if (r.grade.edited_existing_tests?.length) f.edited_existing_tests++; + if (r.grade.edited_test_support?.length) f.edited_test_support++; + } + const dropped = new Set( + rows + .filter( + (r) => + (!o.keepFlagged && r.grade.flags?.excluded) || + (o.errors === 'drop' && r.session.api_error) + ) + .map((r) => r.key) + ); + rows = rows.filter((r) => !dropped.has(r.key)); + if (!rows.some((r) => r.arm === o.base)) + throw new Error( + `No graded ${o.base} sessions left in ${o.runs.join(', ')}${ + dropped.size + ? ` after dropping ${dropped.size} flagged (task, run) pairs: ${[ + ...dropped, + ].join(', ')} (see --keep-flagged)` + : '' + }` + ); + const comparisons = arms.map((arm) => compareArm(rows, arm, o.base)); + const pooled = Object.fromEntries( + comparisons + .filter((c) => PRIMARY.includes(c.arm) && c.build.pairs) + .map((c) => [c.arm, c.build.all_green.p]) + ); + const primary = {}; + for (const c of comparisons.filter((x) => PRIMARY.includes(x.arm))) + for (const category of PRIMARY_CATEGORIES) { + const build = c.byCategory[category]?.build; + if (build) primary[`${category}:${c.arm}`] = build.all_green.p; + } + const report = { + work: o.dir, + runs: o.runs, + split: o.split, + base: o.base, + flags, + droppedPairs: [...dropped].sort(), + holm_primary: holm(primary), + holm_all_green: holm(pooled), + comparisons, + ...(arms.includes('dump10x') && rows.some((r) => r.arm === 'dump') + ? { dump10x_vs_dump: compareArm(rows, 'dump10x', 'dump') } + : {}), + }; + const json = `${JSON.stringify(report, null, 2)}\n`; + if (o.out) await writeFile(resolve(o.out), json); + process.stdout.write(json); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) + main().catch((error) => { + console.error(error.message); + process.exitCode = 1; + }); diff --git a/benchmarks/proof-build/scripts/gate.mjs b/benchmarks/proof-build/scripts/gate.mjs new file mode 100644 index 00000000..ce31af10 --- /dev/null +++ b/benchmarks/proof-build/scripts/gate.mjs @@ -0,0 +1,336 @@ +#!/usr/bin/env node +// The offline gate: free checks that every task must pass before any paid +// session runs (DESIGN.md §4). +// +// node scripts/gate.mjs --hidden