diff --git a/.flatbread-proof/findings/fnd-rules-only-proof-holds-decide-builds-a-suspect-w--kfhw06ex8epckc41.md b/.flatbread-proof/findings/fnd-rules-only-proof-holds-decide-builds-a-suspect-w--kfhw06ex8epckc41.md new file mode 100644 index 00000000..16b2e856 --- /dev/null +++ b/.flatbread-proof/findings/fnd-rules-only-proof-holds-decide-builds-a-suspect-w--kfhw06ex8epckc41.md @@ -0,0 +1,16 @@ +--- +id: fnd-rules-only-proof-holds-decide-builds-a-suspect-w--kfhw06ex8epckc41 +effort: eff-effort-graph-memory-and-agent-wedge--szeqvmgqjqnhd002 +title: Rules only Proof holds decide builds; a suspect warning helps only when read +kind: measurement +created_at: '2026-10-21T15:52:29.557Z' +derives_from: + - fnd-a-stale-decision-costs-builds-more-than-proof-gi--pdrkn9kjkejk52kk +--- + +Two runs on the feature-building benchmark's bed v2. + +- Proof-only-hard tasks on Sonnet 5.5 (6 build x 3 runs, 2 plan x 2): each rule is a product decision recorded only in Proof before the feature existed. Builds all-green: none 3 of 18, skill 15, find 15, findcode 11, brief 11, dump 18 of 18. Every Proof arm beats none after Holm correction (dump p = 0.0005, skill and find 0.002, brief and findcode 0.043); per task, skill and dump win 6 to 0. Plans: 0.88-1.00 against 0.25. Search arms lost where the query did not reach the record (findcode 0 of 3 on the type-scale task, where find got 3 of 3). +- The suspect warning on Haiku (stale and code-visible tasks, 3 runs): build all-green of 27 went from 17 to 20 for findcode and 13 to 18 for brief; skill, find and dump did not change. findcode passed the stale shortcut task 3 of 3 after 0 of 3, because sessions opened the Decision and read the warning; where the stale Decision was peripheral, sessions saw only find's summary line and still followed it (0 of 3); one warning inside a full dump changed nothing. + +Proof's value turns on whether the record is right: a rule the code cannot teach lifts builds from 3 to 11-18 of 18, while an out-of-date record costs the more of Proof is pasted in. The warning recovers that cost only for agents that open the record, so it belongs on every search hit and at the top of pasted context. diff --git a/benchmarks/proof-build/DESIGN.md b/benchmarks/proof-build/DESIGN.md index c97137b5..0311dabe 100644 --- a/benchmarks/proof-build/DESIGN.md +++ b/benchmarks/proof-build/DESIGN.md @@ -2,7 +2,10 @@ Status: pre-registered on 2026-10-20, before any bed, task or paid session exists. Changes after the pilot are allowed only where this file says so, -and each one is logged at the end with its reason. +and each one is logged at the end with its reason. Version 2 (2026-10-21, +after the first full run and before any v2 session) adds the +proof-only-hard category, its records, H5 and its run; the v1 text is kept +and the v2 additions are marked as such. ## Question @@ -26,6 +29,16 @@ Pre-registered hypotheses: all-green. Proof only adds cost there. - **H3 (harm).** On stale tasks, `brief` and `dump` do worse than `none` on all-green. `skill`, `find` and `findcode` do no worse than `none`. +- **H5 (help where the code cannot teach; v2).** On proof-only-hard tasks, + whose rule is a product decision recorded before the feature existed and + hinted at nowhere in the code or its history, `skill`, `find`, + `findcode`, `brief` and `dump` are all-green more often than `none`, and + `none` passes the constraint tests of at most a third of the sessions + that passed acceptance. Plans: on the two proof-only-hard plan tasks, + every Proof arm scores higher than `none`. If `find` and `findcode` gain + less than `brief` and `dump`, that is read as a retrieval miss (the + session did not find the record), which the transcripts' Proof reads + show. - **H4 (burden).** On control tasks, all-green is the same in every arm. Context tokens, processed tokens and wall time rise with the amount of Proof forced in (`none` ≤ `skill` < `find` ≤ `findcode` < `brief` < @@ -78,7 +91,8 @@ as mutations replayed through the real `createProofWriter`, the way Ines Okafor (model, persistence), Tomas Lindqvist (commands, history), Priya Raman (panel, units) and Mateo Alves (layers, keys, plugins). A fifth author, Ren Ito, joins in May and makes the drift commits. About 36 - commits implement a record and carry `Proof: ` trailers. The others + commits implement a record and carry `Proof: ` trailers (37 in v2, + where c34 also carries `plugin.con-content-export`). The others are fixes, tests and chores with ordinary subjects. Commit messages never state a rationale that only Proof holds. - **Variants.** `--variant proof` writes `.flatbread-proof/`, @@ -128,18 +142,60 @@ as mutations replayed through the real `createProofWriter`, the way | V3 | undo restores the exact selection from before the change | yes (the entry shape in `history.ts` and a test for `group`) | | V4 | plugin API v1 is frozen | yes (a header comment and a surface snapshot test) | +- **Planned-work rules (v2, the U rules).** Product decisions recorded + before the feature existed: a request deferred with its agreed behaviour + (Issue resolved `deferred` by an accepted Decision), a Decision accepted + for planned work, a Constraint from a customer, or a Finding from a + usability session. 18 new live records, written at the commits their + authors were working on (all before the first drift commit, so the + drift checks hold), with names of customers (Northwind, Halden, + Brightline) and sessions that appear nowhere else. They may name the + feature's area, as a deferred Issue would; they never use a v2 prompt's + product names (palette titles, the event name), which are banned in + Proof. Only c34, which made `export` the one way for a plugin to read the + document, implements one of them and carries its trailer. Nothing in the + code, tests, docs or commit messages hints at any of them: their wording + is in `BANNED_IN_CODE`, and the gate and `check-bed.mjs` grep the none + bed's whole history for it. The code points the other way where it says + anything: `createFrame` gives a white fill and clipping (U3), the plugin + events go to every plugin alike (U1), `isInteractive` skips hidden and + locked layers together (U4), `walk` covers the whole page (U5). Two + parts are less hidden than the rest. U1: export is already the one way + for a plugin to read the document (`createApi(loaded, canExport)`), and + a commit subject says "Remove getNode from the plugin API — Events + already carry ids", so a careful builder could infer that names stay + behind export; B-H1 is read with that in mind. U8's hidden-layers half: + `walk` skips hidden subtrees, so a plan built from the code leaves them + out too; L-H2's rubric does not score it (see its rubric below). + + | Key | Rule | Records (commit) | + | --- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | + | U1 | layer names and text reach only plugins with `export`; other plugins get the ids alone, in the same event | Constraint from Northwind's security review (c34, trailer) | + | U2 | a text size step goes to the next size on the type scale (8 … 96), not 1 px | Issue from Halden, deferred by an accepted Decision (c45) | + | U3 | a frame made around existing layers has no fill and does not clip | Finding from the February usability sessions, accepted Decision (c11) | + | U4 | an edit that sweeps the whole document skips locked layers and their contents, and changes hidden ones | Issue from Brightline and Halden, deferred by an accepted Decision (c46) | + | U5 | picking similar layers stays inside the starting layer's top-level frame | Finding from the March sessions, accepted Decision (c39) | + | U6 | a whole-pixel fit rounds edges, not sizes, so shapes that touched still touch | Issue, Finding (the prototype's seams), Decision deferring it (c75) | + | U7 | comments live outside the document file, are never history entries, and outlive the layers they are pinned to | Constraint from Halden's contract (c47), Decision (c61), Finding and Decision (c48) | + | U8 | other formats are written by exporter plugins; exports carry no layer names and no hidden layers | Decision and Constraint (c52) | + - **Filler at 10x** (`--scale 10`, used only by `dump10x`). Nine filler Efforts in look-alike areas (export, prototyping, comments, assets, colour management, collaboration cursors, onboarding, telemetry, fonts) - bring the graph to about 1,500 records. They are written in the same + bring the graph to about 1,500 records (1,565 in v2). They are written in the same voice and cite only filler paths that do not exist in the bed. They never state a rule about the seven real areas. The code is identical. - **No anticipation.** Records describe past work and standing rules. An offline check fails the build if any record body, title or commit message contains a task's feature phrases (for example "corner radius" or "marquee"). Area words such as "properties panel" or "undo" are allowed. + The proof-only-hard records are the deliberate exception (v2): they + anticipate their feature, as records of planned work do. What they may + not hold is the prompt's own product names, and the gate checks that no + prompt shares a run of four words with its records or holds a leak + word. -## 3. Tasks (26: 18 build, 8 plan) +## 3. Tasks (34: 24 build, 10 plan; v1 had 26: 18 build, 8 plan) Prompts describe intended behaviour in a user's words. They name no file, symbol, module, Effort or record id. They may name product surface: palette @@ -176,6 +232,26 @@ says what changed from the pre-registered list and why. | B-K2 | control | Fill and Stroke accept `#f80`, six-digit hex with or without `#`, and `rgb(r, g, b)`; shown as upper-case `#RRGGBB` | none | `parseColor` itself widened, which breaks an existing test | | B-K3 | control | Distribute horizontally or vertically with an optional typed gap | none | a missing gap read as 0, which breaks an existing test | +### Proof-only-hard build tasks (6, v2) + +Added after the first full run, which showed that the v1 proof-only rules +are patterns the code already follows (`none` passed 17 of 18), so a +careful builder copies them. Each of these rules is a U rule: a product +decision recorded before the feature existed, which the code does not +hint at and the natural implementation from the prompt breaks. The naive +patch is that natural implementation; it passes acceptance and fails only +constraints. `task.json` gains `records`, the qualified keys of the +records that hold the rule. + +| Id | Request (gist) | Rule | Naive patch | +| ---- | -------------------------------------------------------------------------------------------------------------------- | ---- | --------------------------------------------------------------------- | +| B-H1 | A `layerrename` plugin event, `{ id, before, after }`, from the palette, the layers panel, undo and redo | U1 | the payload goes to every plugin, as the other events do | +| B-H2 | Increase font size and Decrease font size in the palette, a step at a time, for every selected text layer; undoable | U2 | ±1 px per step | +| B-H3 | Frame selection: a new frame drawn tightly around the selection, in the frontmost one's place, selected; one Undo | U3 | the frame takes `createFrame`'s white fill and clipping | +| B-H4 | Replace colour: type two colours; every fill and stroke in the first gets the second, whatever is selected; one Undo | U4 | every layer changes, locked ones included | +| B-H5 | Select same fill: from one layer, select the layers whose fill is the same colour, inside groups too | U5 | searches the whole page | +| B-H6 | Round to pixels: X, Y, W and H of the selection, and of what is inside selected groups and frames, become whole | U6 | rounds x, y, width and height each on its own, which opens 1 px seams | + ### Plan tasks (8) The agent writes `PLAN.md` and changes no code. Each plan task needs at @@ -192,6 +268,23 @@ least two rules and is scored on a rubric (§5). | L-K1 | control | Rulers along the top and left with nicely spaced labelled ticks for the current zoom | none | | L-K2 | control | A document swatches list built from the fills in use, sorted by frequency | none | +### Proof-only-hard plan tasks (2, v2) + +| Id | Request (gist) | What it hinges on | +| ---- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| L-H1 | Review comments: pin to a point or a layer, threads, resolve and reopen; plugins hear of new ones | U7 (outside the document file, no history entries, pins outlive layers), U1 (comment text only for plugins with `export`), V4 | +| L-H2 | Export a frame as SVG to hand to developers | U8 (an exporter plugin on the export capability, no layer names, generated ids), V4 | + +Their plausible plans (written from the code alone) store comments in the +file through commands that Undo can remove and delete them with their +layer, and put an SVG module in core that names elements after layers: +they should score low on every U point. L-H2's rubric scores only what the +code cannot teach: where the exporter lives, no layer names, generated +ids, and V4. Leaving hidden layers out and placing layers relative to the +frame follow from the code (`walk`, absolute positions), so they are not +points; the plausible plan does both, and gets V4 (a visible rule) and no +U point. + ## 4. Grading build tasks 1. **Capture.** When a session ends, the runner writes `session.diff`: @@ -219,8 +312,8 @@ least two rules and is scored on a rubric (§5). - **Constraints:** `constraints.test.ts` plus the global invariants. `global.test.ts` runs on every task: a document saves, loads and saves the same text, and plugin API v1 keeps its members and shortcut table - (both stated in the code). `global-rules.test.ts` runs on proof-only - and stale tasks only: every shipped migration step returns what it + (both stated in the code). `global-rules.test.ts` runs on proof-only, + proof-only-hard and stale tasks only: every shipped migration step returns what it always returned, and every palette command makes at most one history entry per run. Those are R3 and R1, which only Proof states, so a control or code-visible task is never graded on them. @@ -253,6 +346,10 @@ it must pass for every task): - `leakWords` for each R rule appear nowhere in the none variant's code, tests, docs or commit messages. The phrases for each V rule do appear. - `featureWords` appear nowhere in Proof. +- Proof-only-hard (v2): the same two checks, plus: the task's `records` + are live in the proof bed; every leak word is in them (the rule is in + Proof); and the prompt holds no leak word and shares no run of four + words with them (it does not echo Proof). - The two variants' code trees are identical. ## 5. Grading plan tasks @@ -320,7 +417,7 @@ clone's own graph. | `find` | as `skill` | "Before planning, run `./flatbread proof find --all`, then `proof get` any hit that bears on the request." | | `findcode` | as `skill` | the `find` text plus: "Before editing a file, run `./flatbread proof find --path `; before relying on a Decision, run `proof find --refs all`." | | `dump` | as `skill` | every live record body (all kinds, compactions included), grouped by Effort | -| `dump10x` | as `skill`, on the 10x graph | as `dump`, 1,074 live records of 1,547 (the burden ceiling; an extra arm, not a primary comparison) | +| `dump10x` | as `skill`, on the 10x graph | as `dump`, 1,074 live records of 1,547 (1,092 of 1,565 in v2; the burden ceiling; an extra arm, not a primary comparison) | `none` has no `CLAUDE.md` line about Proof, no skill, no wrapper, no config and no node_modules link. The proof arms link node_modules (for the @@ -358,7 +455,12 @@ change code." α = 0.05 (`holm_primary` in `compare.mjs`). The stricter task-level sign test, which counts each task once as in `proof-recall/scripts/compare.mjs`, is reported beside each. `dump10x` is compared with `dump`, without - correction. + correction. v2 adds all-green on the proof-only-hard build tasks (H5, + 6 × 3 = 18 pairs per arm) as a third primary category, so a v2 report + Holm-corrects fifteen tests; a run without proof-only-hard tasks keeps + the original ten. The H5 constraint share (`none` constraint passes among + its acceptance passes) and the plan scores on L-H1 and L-H2 are + secondary. - **Secondary.** All-green pooled over all 18 build tasks, Holm-corrected across the five arms (`holm_all_green`); it mixes categories whose predicted effects run in opposite directions, so it is read only with @@ -434,6 +536,24 @@ tokens) for `dump10x`. sessions about $30 to $40. Judging covers 104 plans, twice each, for about $14. The total is about $65 to $85. At 8 parallel jobs and about 8 minutes per session, that is about 7.5 hours of wall time. +- **v2 run on Sonnet** (pre-registered before any v2 session). The eight + proof-only-hard tasks on `claude-sonnet-5-5`, rather than Haiku, in the + six primary arms: build tasks 3 runs, plan tasks 2, `dump10x` once per + build task (L-H1 and L-H2 skip it, see the change log). 6 × 6 × 3 = 108 build and + 2 × 6 × 2 = 24 plan sessions, plus 6 `dump10x`, 138 in all. At about $0.90 for a `none` build session (as the + Sonnet subset estimate), $1.20 with `brief`, $1.50 with `dump`, $0.40 for + a plan and about $6 for `dump10x` (more if long-prompt rates apply), that + is about $120 for build, $10 for plan and $36 for `dump10x`, with about + $4 of judging (48 calls) and $1 for the rubric gate on the six new + plans: about $170. The 26 v1 tasks keep their Haiku results and are not + rerun; the v2 none bed is byte-identical to v1's (same tip, `bdb60f4`), + and the gate grades their patches as it did. The v2 pilot is B-H1 and + B-H4 in the six primary arms, once (12 sessions, about $13), with the + rubric gate on L-H1 and L-H2; it must clear the same gates as the v1 + pilot, and in addition `find` and `findcode` must surface the task's + rule record in at least one of their two sessions, or the prompts and + records are revised (the only pilot change allowed for v2 besides the + v1 list). - **Sonnet subset** (the 8 build tasks P1, P3, P5, P7, V1, S1, S3 and K1; the arms `none`, `skill`, `find` and `brief`; 1 run). 32 sessions at about $0.90 each, for about $30. It is read only as a check on the @@ -451,6 +571,9 @@ tokens) for `dump10x`. - `spec/filler.mjs`: the 10x filler. - `tasks//`: each task's `task.json`, and for plan tasks `rubric.json` and `plans/`. +- `hidden//`: hidden tests and the reference and naive patches, and + `hidden/_global/` (committed since the first full run; copied to + `/hidden` before a run). - `scripts/` (`gen-bed.mjs`, `check-bed.mjs`, `gate.mjs`, `run.mjs`, `grade.mjs`, `judge.mjs`, `compare.mjs`, with `arms.mjs`, `inject.mjs` and `common.mjs`), reusing `proof-recall`'s `collect.mjs` and its sign test. @@ -596,3 +719,60 @@ tokens) for `dump10x`. (B-P1, B-S3, B-V4, B-K1) in the six primary arms, 24 sessions; L-P1 and `dump10x` are optional add-ons. The `dump10x` arm text gives its real size: 1,074 live records of the 1,547 record files. +- 2026-10-21, v2, after the first full run (446 Haiku sessions) and before + any v2 session: + - **Why.** `none` passed 17 of 18 proof-only build sessions: one undo + step per gesture, untouched shipped migrations and frozen plugin + payloads are patterns the code already follows, so the benchmark could + not show a benefit even if Proof has one. + - **New category, proof-only-hard:** six build tasks (B-H1 to B-H6) and + two plan tasks (L-H1, L-H2) on the U rules (§2), with H5 and a third + primary category (§7). Counts are now build 6/5/4/3/6 and plan + 3/1/2/2/2. + - **Records.** 18 live records in `spec/proof.mjs` (`PLANNED`), merged + into the commits c11, c34, c39, c45, c46, c47, c48, c52, c61 and c75; + one trailer (c34). The none bed is unchanged (tip `bdb60f4`); the proof + beds are rebuilt (bed `39c5772`, bed-x10 `56d814d`), the same hashes + on a second build, and `check-bed.mjs` passes its 33 checks. + - **Banned words.** The new prompts' product names are in + `BANNED_IN_PROOF` and the U rules' wording in `BANNED_IN_CODE`; the old + lists are unchanged. + - **Gate.** The proof-only-hard checks of §4 are new; the old + categories' checks are unchanged. On bed v2 the gate passes 193 static + checks and 72 graded patches, and the 54 graded v1 patches give the + same results, failing tests included, as on bed v1. + - **Grading.** `global-rules.test.ts` also runs with proof-only-hard + tasks; `judge.mjs --gate` reports plausible plans of proof-only-hard + tasks as it does for proof-only ones. + - **Model.** The v2 run is pre-registered on Sonnet (§8). +- 2026-10-21, v2, after review and before any v2 session: + - **B-H4 prompt.** "Every fill and stroke in the first colour then gets + the second, wherever the layer sits" and "everywhere in a document" + told the agent to change locked layers too, so a Proof session that + found U4 had to overrule the person. It now reads "across a document" + and "Fills and strokes in the first colour then get the second, in + groups and frames too, whatever is selected". + - **B-H1 prompt.** The payload is now what plugin authors "have asked + for", not a fixed spec, for the same reason in a milder form; §2 notes + that the code half-teaches U1. + - **B-H2.** The prompt asks for one Undo per step ("one Undo takes a + whole step back") and the acceptance test checks one history entry, + so a step that is not one batch fails acceptance, not R1's global + test, and H5's constraint share counts only U2. + - **L-H2.** "with its layers drawn as they look on the canvas" is gone + from the prompt, and the rubric's hidden-layers and frame-relative + points (include) and hidden-layers point (avoid) are replaced by + "generated ids" (4 includes, 2 avoids): those points were reachable + from the code. + - **dump10x.** L-H1 and L-H2 skip `dump10x` (`skipArms` in task.json, + honoured by `run.mjs`): the 10x filler has Efforts on comments and on + export (`fx-comments`, `fx-export`), so on those two tasks `dump10x` + against `dump` would measure conflicting filler, not scale. The v2 run + is 138 sessions, about $170 (§8). + - **SUSPECT warning (bed v2).** After the suspect rule was revised, the + proof bed flags 5 records, none of them a v2 record. One rests on V4: + `dec-freeze-plugin-api-v1` reads "rests on records that are no longer + live" (its survey Finding was invalidated by the c55 recount). That is + the bed's own state since v1 and is left as it is; it tells the agent + to check what replaced the Finding, not to follow the code. The false + "the code wins" warning on `con-stored-geometry` (V2) is gone. diff --git a/benchmarks/proof-build/README.md b/benchmarks/proof-build/README.md index ed2f1b2a..28004958 100644 --- a/benchmarks/proof-build/README.md +++ b/benchmarks/proof-build/README.md @@ -14,15 +14,30 @@ alternatives, Constraints, Findings, Risks, code Citations) sit in the graph. The none variant has the same code and no Proof at all. The 10x variant adds 9 filler Efforts, about ten times the records. -There are 26 tasks: 18 build tasks graded by hidden tests and 8 plan tasks -scored by a blind rubric. Each task belongs to one category: - -| Category | What makes a solution right | Build | Plan | -| ------------ | ----------------------------------------------------- | ----- | ---- | -| proof-only | a rule that only Proof states | 6 | 3 | -| code-visible | a rule Proof states and the code also shows | 5 | 1 | -| stale | the code, where Proof is out of date | 4 | 2 | -| control | nothing about Proof; measures the cost of carrying it | 3 | 2 | +There are 34 tasks: 24 build tasks graded by hidden tests and 10 plan +tasks scored by a blind rubric. Each task belongs to one category: + +| Category | What makes a solution right | Build | Plan | +| --------------- | --------------------------------------------------------------------------------- | ----- | ---- | +| proof-only | a rule that only Proof states (the code's patterns turned out to teach it) | 6 | 3 | +| proof-only-hard | a product decision Proof recorded before the feature existed; nothing hints at it | 6 | 2 | +| code-visible | a rule Proof states and the code also shows | 5 | 1 | +| stale | the code, where Proof is out of date | 4 | 2 | +| control | nothing about Proof; measures the cost of carrying it | 3 | 2 | + +The proof-only-hard tasks were added for v2 of the bed, after the first full +run showed that the proof-only rules can be copied from the code: + +| Id | Kind | Request (gist) | The rule only Proof holds | +| ---- | ----- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| B-H1 | build | a `layerrename` plugin event with `{ id, before, after }` | layer names reach only plugins with the `export` capability; the others get `{ id }` (a customer's security review) | +| B-H2 | build | Increase font size and Decrease font size in the palette | a step goes to the next size on the type scale (16 to 18, 13 to 14), not 1 px (a deferred customer request) | +| B-H3 | build | Frame selection: wrap the selection in a new frame that fits it | the new frame has no fill and does not clip (a usability finding: the white fill hid layers) | +| B-H4 | build | Replace colour: swap one colour for another across the document | locked layers and their contents keep their colour; hidden layers change (agreed with two customers) | +| B-H5 | build | Select same fill: select the layers with the selected layer's fill colour | the search stays inside the selected layer's top-level frame (a usability finding: edits on screens nobody saw) | +| B-H6 | build | Round to pixels: X, Y, W and H of the selection become whole numbers | edges are rounded, not sizes, so shapes that touched still touch (a finding on the prototype's seams) | +| L-H1 | plan | review comments pinned to points or layers, with threads, resolving and a plugin event | comments live outside the document file, are never undo steps, outlive their layer; comment text only for `export` plugins | +| L-H2 | plan | export a frame as SVG for developers | exporters are plugins on the export capability; no layer names and no hidden layers in the output | Every session gets the same code and the same request. The arms differ in how Proof reaches the agent: @@ -150,6 +165,29 @@ done node scripts/compare.mjs $P4/runs --runs r1,r2,r3 --out $P4/runs/compare.json ``` +### v2: the proof-only-hard tasks on Sonnet + +Pre-registered in DESIGN.md §8: the eight proof-only-hard tasks on +`claude-sonnet-5-5` in the six primary arms (build 3 runs, plan 2, +`dump10x` once per build task), 138 sessions, about $170 with judging. +L-H1 and L-H2 skip `dump10x` (`skipArms` in their task.json), since the +10x filler has Efforts on comments and on export. Rebuild the beds +first (the proof beds changed in v2; the none bed did not), pilot on B-H1 +and B-H4, then: + +```sh +node scripts/run.mjs $P4/runs --name h1 --bed $P4/bed --only proof-only-hard \ + --model claude-sonnet-5-5 --jobs 8 +node scripts/run.mjs $P4/runs --name h2 --bed $P4/bed --only proof-only-hard \ + --model claude-sonnet-5-5 --jobs 8 +node scripts/run.mjs $P4/runs --name h3 --bed $P4/bed --only proof-only-hard \ + --ids B-H1,B-H2,B-H3,B-H4,B-H5,B-H6 --model claude-sonnet-5-5 --jobs 8 +node scripts/run.mjs $P4/runs --name h1 --bed $P4/bed --only proof-only-hard \ + --arms dump10x --model claude-sonnet-5-5 --jobs 4 +``` + +then grade, judge and compare `h1,h2,h3` as above. + ## Results: Haiku, full run (2026-10-21) 18 build tasks × 3 runs and 8 plan tasks × 2 runs in each of the six primary @@ -224,6 +262,76 @@ teaches a builder most of them. The changes this points to are a read that marks a Decision itself as suspect when the code it rests on has moved, and proof-only tasks whose rules the code does not hint at. +## Results: v2 (2026-10-21) + +Two runs on bed v2. Per-session rows and `compare.mjs` output are in +`results/` (`sonnet-hard-*`, `haiku-suspect-*`). + +### Proof-only-hard tasks on Sonnet 5.5 + +6 build tasks × 3 runs and 2 plan tasks × 2 runs in the six primary arms, +`dump10x` once per build task except B-H1, plus a 12-session pilot: 149 +sessions on `claude-sonnet-5-5`. No session timed out; one `none` session +was flagged and its pair dropped. + +| Arm | B-H1 | B-H2 | B-H3 | B-H4 | B-H5 | B-H6 | builds all-green | plans (L-H1, L-H2) | context tokens | +| ---------- | ---- | ---- | ---- | ---- | ---- | ---- | ---------------- | ------------------ | -------------- | +| `none` | 0/3 | 0/3 | 2/3 | 0/3 | 0/3 | 1/3 | **3 of 18** | 0.25, 0.25 | 20k | +| `skill` | 3/3 | 3/3 | 3/3 | 2/3 | 1/3 | 3/3 | 15 of 18 | 0.88, 1.00 | 24k | +| `find` | 3/3 | 3/3 | 3/3 | 3/3 | 0/3 | 3/3 | 15 of 18 | 1.00, 1.00 | 26k | +| `findcode` | 3/3 | 0/3 | 2/3 | 3/3 | 0/3 | 3/3 | 11 of 18 | 1.00, 1.00 | 26k | +| `brief` | 1/3 | 2/3 | 1/3 | 2/3 | 2/3 | 3/3 | 11 of 18 | 1.00, 1.00 | 34k | +| `dump` | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | **18 of 18** | 0.94, 1.00 | 59k | +| `dump10x` | - | 1/1 | 1/1 | 1/1 | 1/1 | 1/1 | 5 of 5 | - | 239k | + +- **When the rule lives only in Proof, Proof decides the outcome.** Without it + Sonnet built the feature and broke the recorded rule (names sent to every + plugin, 1 px font steps, locked layers recoloured, a page-wide search). + Every Proof arm beats `none` on the pre-registered primary test after Holm + correction: `dump` p = 0.0005, `skill` and `find` 0.002, `brief` and + `findcode` 0.043. Counted per task, `skill` and `dump` win 6 tasks to 0 + (p = 0.031). +- **Pasting every record wins here** (18 of 18), the reverse of the stale + tasks, where it did worst. Search arms miss when the query does not reach + the record: `findcode` searched "increase decrease font size palette" and + never met the type-scale Issue (0 of 3 on B-H2, where `find` got 3 of 3), and + B-H5's frame rule was found by few arms. +- **Plans:** every Proof arm scores 0.88 to 1.00 against 0.25 for `none` + (Wilcoxon p = 0.25 with four pairs). + +### The suspect warning on Haiku + +The stale and code-visible tasks (9 build × 3 runs, 3 plan × 2 runs) in the +six primary arms on bed v2 with the suspect reads: 216 sessions on +`claude-haiku-5-5`, set against v1 (old reads, bed v1). Build all-green of +27: + +| Arm | v1 (old reads) | v2 (suspect reads) | B-S3 v1 → v2 | +| ---------- | -------------- | ------------------ | ------------ | +| `none` | 24 | 23 | 3/3 → 3/3 | +| `skill` | 17 | 17 | 0/3 → 1/3 | +| `find` | 16 | 16 | 0/3 → 1/3 | +| `findcode` | 17 | **20** | 0/3 → 3/3 | +| `brief` | 13 | **18** | 0/3 → 2/3 | +| `dump` | 12 | 12 | 0/3 → 0/3 | + +- **It works when the agent reads the record.** `findcode` sessions on B-S3 + opened the stale Decision, read the warning, followed the code and passed + all three runs; `findcode` stale tasks went from 9 to 12 of 12. +- **It does not when the record is peripheral.** On B-V4, where the shortcut + is a side detail, sessions saw only the summary line of `find` and still + edited the frozen table (0 of 3). +- **Pasted in full, it is lost.** One warning inside 89k characters of + records changed nothing for `dump`, and one stale plan got worse (L-S1 + `dump` 0.64 to 0.29). +- Bed v2 carries 18 more records than v1, so brief and dump are slightly + larger; the none bed is the same. + +So Proof's value turns on whether its record is right. Where it holds a rule +the code cannot teach, any way of reading it lifts builds from 3 to 11–18 of +18; where it is out of date, the more of it is pasted in, the worse builds +get. The warning recovers the cost only for agents that open the record. + ## Layout | Path | What it holds | @@ -235,6 +343,7 @@ proof-only tasks whose rules the code does not hint at. | `spec/filler.mjs` | the 9 filler Efforts of the 10x graph | | `spec/banned.mjs` | words no record or commit may hold, so Proof never names a task | | `tasks//` | `task.json`; plan tasks also `rubric.json` and `plans/` | +| `hidden//` | hidden tests and reference and naive patches; `hidden/_global/` invariants | | `scripts/gen-bed.mjs` | builds a bed | | `scripts/check-bed.mjs` | checks built beds against each other and the design | | `scripts/gate.mjs` | the offline gate | @@ -243,6 +352,8 @@ proof-only tasks whose rules the code does not hint at. | `scripts/judge.mjs` | blinds and judges plans; the rubric gate | | `scripts/compare.mjs` | compares each arm with `none` | -Hidden tests and the reference and naive patches are in -`/hidden//`, never in this repository. `FLATBREAD_PROOF_SERVER` -must stay unset; Proof reads run locally against each clone's own graph. +Hidden tests and the reference and naive patches are in `hidden//` +(committed since the first full run) and are copied to `/hidden//` +for a run, so no session needs a path into this repository. +`FLATBREAD_PROOF_SERVER` must stay unset; Proof reads run locally against +each clone's own graph. diff --git a/benchmarks/proof-build/hidden/B-H1/acceptance.test.ts b/benchmarks/proof-build/hidden/B-H1/acceptance.test.ts new file mode 100644 index 00000000..db226a90 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H1/acceptance.test.ts @@ -0,0 +1,76 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// Every plugin here has the export capability, so it may read names. +function setup() { + const e = createEditor(); + const card = e.create('rect', { name: 'Card' }); + const badge = e.create('rect', { name: 'Badge' }); + const got: any[] = []; + e.plugins.load({ + name: 'watcher', + capabilities: ['export'], + main(api) { + api.on('layerrename' as never, (p: any) => got.push(p)); + }, + }); + const seen = () => + got.map((p) => ({ id: p.id, before: p.before, after: p.after })); + return { e, card, badge, got, seen }; +} + +test('layerrename is a plugin event', () => { + const { e } = setup(); + assert.deepEqual(e.plugins.errors(), []); +}); + +test('renaming from the palette tells the plugin the old and new names', () => { + const { e, card, seen } = setup(); + e.selection.set([card]); + e.palette.run('Rename layer', 'Hero'); + assert.deepEqual(seen(), [{ id: card, before: 'Card', after: 'Hero' }]); +}); + +test('renaming in the layers panel fires it too', () => { + const { e, badge, seen } = setup(); + assert.ok(e.layers.rename(badge, 'Price tag')); + assert.deepEqual(seen(), [ + { id: badge, before: 'Badge', after: 'Price tag' }, + ]); +}); + +test('undo and redo of a rename fire it', () => { + const { e, card, seen } = setup(); + e.selection.set([card]); + e.palette.run('Rename layer', 'Hero'); + assert.ok(e.history.undo()); + assert.ok(e.history.redo()); + assert.deepEqual(seen(), [ + { id: card, before: 'Card', after: 'Hero' }, + { id: card, before: 'Hero', after: 'Card' }, + { id: card, before: 'Card', after: 'Hero' }, + ]); +}); + +test('other edits and an unchanged name do not fire it', () => { + const { e, card, badge, got } = setup(); + e.selection.set([card, badge]); + e.panel.edit('X', '40'); + e.panel.edit('Fill', '#336699'); + e.palette.run('Hide'); + assert.equal(e.layers.rename(card, 'Card'), false); + assert.equal(got.length, 0); +}); + +test('each rename of several layers is reported once', () => { + const { e, card, badge, seen } = setup(); + e.layers.rename(card, 'One'); + e.layers.rename(badge, 'Two'); + e.layers.rename(card, 'Three'); + assert.deepEqual(seen(), [ + { id: card, before: 'Card', after: 'One' }, + { id: badge, before: 'Badge', after: 'Two' }, + { id: card, before: 'One', after: 'Three' }, + ]); +}); diff --git a/benchmarks/proof-build/hidden/B-H1/constraints.test.ts b/benchmarks/proof-build/hidden/B-H1/constraints.test.ts new file mode 100644 index 00000000..0ebbc4a2 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H1/constraints.test.ts @@ -0,0 +1,39 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// U1: layer names and text reach only plugins granted export; every other +// plugin hears of the same event by id alone. +test('a plugin without export hears of a rename by id alone', () => { + const e = createEditor(); + const layer = e.create('text', { name: 'Project Falcon', text: 'Q3 draft' }); + const plain: unknown[] = []; + const cleared: unknown[] = []; + e.plugins.load({ + name: 'plain', + main(api) { + api.on('layerrename' as never, (p: unknown) => plain.push(p)); + }, + }); + e.plugins.load({ + name: 'cleared', + capabilities: ['export'], + main(api) { + api.on('layerrename' as never, (p: unknown) => cleared.push(p)); + }, + }); + // Only a build that has the event can break the rule. + if (e.plugins.errors().length) return; + e.selection.set([layer]); + e.palette.run('Rename layer', 'Project Heron'); + if (!cleared.length) return; + e.history.undo(); + e.layers.rename(layer, 'Project Osprey'); + assert.equal(plain.length, 3, 'the plugin without export still hears of it'); + for (const payload of plain) { + assert.equal((payload as { id?: unknown }).id, layer); + const text = JSON.stringify(payload); + for (const word of ['Falcon', 'Heron', 'Osprey']) + assert.ok(!text.includes(word), `"${word}" reached ${text}`); + } +}); diff --git a/benchmarks/proof-build/hidden/B-H1/naive.patch b/benchmarks/proof-build/hidden/B-H1/naive.patch new file mode 100644 index 00000000..686f34de --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H1/naive.patch @@ -0,0 +1,76 @@ +diff --git a/docs/plugins.md b/docs/plugins.md +index 34a6933..4574baa 100644 +--- a/docs/plugins.md ++++ b/docs/plugins.md +@@ -31,11 +31,12 @@ Plugin API v1 is frozen. + + ## Events + +-| Event | Payload | +-| ---------------- | ---------------------- | +-| `documentchange` | `{ ids }` | +-| `documentload` | `{ name }` | +-| `historychange` | `{ canUndo, canRedo }` | ++| Event | Payload | ++| ---------------- | ----------------------- | ++| `documentchange` | `{ ids }` | ++| `documentload` | `{ name }` | ++| `historychange` | `{ canUndo, canRedo }` | ++| `layerrename` | `{ id, before, after }` | + + ## Capabilities + +diff --git a/src/editor.ts b/src/editor.ts +index c964d1d..b6f9cdb 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -92,6 +92,28 @@ export function createEditor(options: EditorOptions = {}) { + doc.events.on('load', ({ name }) => + plugins.emit('documentload', Object.freeze({ name })) + ); ++ // The name each layer had when plugins last heard of it. ++ const names = new Map(); ++ doc.events.on('change', ({ ids: changed }) => { ++ for (const id of changed) { ++ const node = doc.get(id); ++ if (!node) { ++ names.delete(id); ++ continue; ++ } ++ const before = names.get(id); ++ names.set(id, node.name); ++ if (before !== undefined && before !== node.name) ++ plugins.emit( ++ 'layerrename', ++ Object.freeze({ id, before, after: node.name }) ++ ); ++ } ++ }); ++ doc.events.on('load', () => { ++ names.clear(); ++ for (const node of doc.all()) names.set(node.id, node.name); ++ }); + history.events.on('change', ({ canUndo, canRedo }) => + plugins.emit('historychange', Object.freeze({ canUndo, canRedo })) + ); +diff --git a/src/plugin/events.ts b/src/plugin/events.ts +index ef5b406..53973d3 100644 +--- a/src/plugin/events.ts ++++ b/src/plugin/events.ts +@@ -7,6 +7,8 @@ export interface PluginEvents { + documentload: { name: string }; + /** Undo or redo availability changed. */ + historychange: { canUndo: boolean; canRedo: boolean }; ++ /** A layer's name changed. */ ++ layerrename: { id: string; before: string; after: string }; + } + + export type PluginEventName = keyof PluginEvents; +@@ -15,6 +17,7 @@ export const PLUGIN_EVENTS: readonly PluginEventName[] = [ + 'documentchange', + 'documentload', + 'historychange', ++ 'layerrename', + ]; + + export function isPluginEvent(name: string): name is PluginEventName { diff --git a/benchmarks/proof-build/hidden/B-H1/reference.patch b/benchmarks/proof-build/hidden/B-H1/reference.patch new file mode 100644 index 00000000..bc5cbda3 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H1/reference.patch @@ -0,0 +1,141 @@ +diff --git a/docs/plugins.md b/docs/plugins.md +index 34a6933..75c1639 100644 +--- a/docs/plugins.md ++++ b/docs/plugins.md +@@ -31,11 +31,15 @@ Plugin API v1 is frozen. + + ## Events + +-| Event | Payload | +-| ---------------- | ---------------------- | +-| `documentchange` | `{ ids }` | +-| `documentload` | `{ name }` | +-| `historychange` | `{ canUndo, canRedo }` | ++| Event | Payload | ++| ---------------- | ----------------------- | ++| `documentchange` | `{ ids }` | ++| `documentload` | `{ name }` | ++| `historychange` | `{ canUndo, canRedo }` | ++| `layerrename` | `{ id, before, after }` | ++ ++`layerrename` carries the old and new names only to plugins with the ++`export` capability; other plugins get `{ id }`. + + ## Capabilities + +diff --git a/src/editor.ts b/src/editor.ts +index c964d1d..aa2b309 100644 +--- a/src/editor.ts ++++ b/src/editor.ts +@@ -92,6 +92,30 @@ export function createEditor(options: EditorOptions = {}) { + doc.events.on('load', ({ name }) => + plugins.emit('documentload', Object.freeze({ name })) + ); ++ // Layer names as plugins last heard them, so a rename from anywhere ++ // (palette, layers panel, undo, redo) is seen once, with its old name. ++ const names = new Map(); ++ doc.events.on('change', ({ ids: changed }) => { ++ for (const id of changed) { ++ const node = doc.get(id); ++ if (!node) { ++ names.delete(id); ++ continue; ++ } ++ const before = names.get(id); ++ names.set(id, node.name); ++ if (before !== undefined && before !== node.name) ++ plugins.emitContent( ++ 'layerrename', ++ Object.freeze({ id, before, after: node.name }), ++ Object.freeze({ id }) ++ ); ++ } ++ }); ++ doc.events.on('load', () => { ++ names.clear(); ++ for (const node of doc.all()) names.set(node.id, node.name); ++ }); + history.events.on('change', ({ canUndo, canRedo }) => + plugins.emit('historychange', Object.freeze({ canUndo, canRedo })) + ); +diff --git a/src/plugin/events.ts b/src/plugin/events.ts +index ef5b406..3b384e8 100644 +--- a/src/plugin/events.ts ++++ b/src/plugin/events.ts +@@ -7,6 +7,11 @@ export interface PluginEvents { + documentload: { name: string }; + /** Undo or redo availability changed. */ + historychange: { canUndo: boolean; canRedo: boolean }; ++ /** ++ * A layer's name changed. The names go only to plugins with the `export` ++ * capability; other plugins get the id alone. ++ */ ++ layerrename: { id: string; before?: string; after?: string }; + } + + export type PluginEventName = keyof PluginEvents; +@@ -15,6 +20,7 @@ export const PLUGIN_EVENTS: readonly PluginEventName[] = [ + 'documentchange', + 'documentload', + 'historychange', ++ 'layerrename', + ]; + + export function isPluginEvent(name: string): name is PluginEventName { +diff --git a/src/plugin/host.ts b/src/plugin/host.ts +index 9e82c76..0815d02 100644 +--- a/src/plugin/host.ts ++++ b/src/plugin/host.ts +@@ -20,6 +20,7 @@ type AnyListener = (payload: never) => void; + interface Loaded { + manifest: PluginManifest; + listeners: Map>; ++ canExport: boolean; + } + + export class PluginHost { +@@ -39,10 +40,14 @@ export class PluginHost { + if (this.plugins.has(manifest.name)) + fail('duplicate-plugin', `plugin ${manifest.name} is already loaded`); + const capabilities = checkCapabilities(manifest.capabilities); +- const loaded: Loaded = { manifest, listeners: new Map() }; ++ const loaded: Loaded = { ++ manifest, ++ listeners: new Map(), ++ canExport: capabilities.includes('export'), ++ }; + this.plugins.set(manifest.name, loaded); + this.sandbox.restore(manifest.name); +- const api = this.createApi(loaded, capabilities.includes('export')); ++ const api = this.createApi(loaded, loaded.canExport); + this.sandbox.call(manifest.name, 'main', () => manifest.main(api)); + return api; + } +@@ -58,11 +63,26 @@ export class PluginHost { + } + + emit(event: K, payload: PluginEvents[K]): void { +- for (const [name, plugin] of this.plugins) ++ this.emitContent(event, payload, payload); ++ } ++ ++ /** ++ * Sends `full` to plugins with the `export` capability and `bare` to the ++ * rest. Layer names and text are document content, which only plugins ++ * granted export may read. ++ */ ++ emitContent( ++ event: K, ++ full: PluginEvents[K], ++ bare: PluginEvents[K] ++ ): void { ++ for (const [name, plugin] of this.plugins) { ++ const payload = plugin.canExport ? full : bare; + for (const listener of [...(plugin.listeners.get(event) ?? [])]) + this.sandbox.call(name, event, () => + (listener as (p: PluginEvents[K]) => void)(payload) + ); ++ } + } + + listenerCount(event: PluginEventName): number { diff --git a/benchmarks/proof-build/hidden/B-H2/acceptance.test.ts b/benchmarks/proof-build/hidden/B-H2/acceptance.test.ts new file mode 100644 index 00000000..bc3ba7d2 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H2/acceptance.test.ts @@ -0,0 +1,93 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const title = e.create('text', { + name: 'Title', + text: 'A heading that wraps onto more than one line', + width: 160, + fontSize: 24, + } as never); + const body = e.create('text', { + name: 'Body', + text: 'Short', + x: 200, + fontSize: 12, + } as never); + const box = e.create('rect', { name: 'Box', x: 400 }); + return { e, title, body, box }; +} +type Editor = ReturnType; +const size = (e: Editor, id: string) => (e.doc.require(id) as any).fontSize; + +/** The height the text box gets when that font size is typed into the panel. */ +function typedHeight(e: Editor, id: string, fontSize: number): number { + const twin = createEditor(); + const { text, width, lineHeight } = e.doc.require(id) as any; + const copy = twin.create('text', { text, width, lineHeight } as never); + twin.selection.set([copy]); + twin.panel.edit('Font size', String(fontSize)); + return twin.doc.require(copy).height; +} + +test('both commands are offered only when the selection has a text layer', () => { + const { e, title, box } = setup(); + for (const ids of [[], [box]]) { + e.selection.set(ids); + assert.ok(!e.palette.list().includes('Increase font size')); + assert.ok(!e.palette.list().includes('Decrease font size')); + } + for (const ids of [[title], [title, box]]) { + e.selection.set(ids); + assert.ok(e.palette.list().includes('Increase font size')); + assert.ok(e.palette.list().includes('Decrease font size')); + } +}); + +test('Increase font size makes each selected text layer bigger, from its own size', () => { + const { e, title, body, box } = setup(); + const boxBefore = structuredClone(e.doc.require(box)); + e.selection.set([title, body, box]); + assert.ok(e.palette.run('Increase font size')); + assert.ok(size(e, title) > 24 && size(e, title) <= 48); + assert.ok(size(e, body) > 12 && size(e, body) <= 24); + assert.deepEqual(e.doc.require(box), boxBefore); +}); + +test('Decrease font size makes each selected text layer smaller', () => { + const { e, title, body } = setup(); + e.selection.set([title, body]); + assert.ok(e.palette.run('Decrease font size')); + assert.ok(size(e, title) < 24 && size(e, title) >= 12); + assert.ok(size(e, body) < 12 && size(e, body) >= 6); +}); + +test('the text box fits the new size as it does for a typed size', () => { + const { e, title, body } = setup(); + e.selection.set([title, body]); + e.palette.run('Increase font size'); + for (const id of [title, body]) + assert.equal(e.doc.require(id).height, typedHeight(e, id, size(e, id))); + e.palette.run('Decrease font size'); + e.palette.run('Decrease font size'); + for (const id of [title, body]) + assert.equal(e.doc.require(id).height, typedHeight(e, id, size(e, id))); + e.selection.set([title]); + assert.equal(e.panel.value('Font size'), String(size(e, title))); +}); + +test('one Undo takes a whole step back', () => { + const { e, title, body } = setup(); + const before = [title, body].map((id) => structuredClone(e.doc.require(id))); + e.selection.set([title, body]); + const entries = e.history.size; + e.palette.run('Increase font size'); + assert.equal(e.history.size, entries + 1); + assert.ok(e.history.undo()); + assert.deepEqual( + [title, body].map((id) => e.doc.require(id)), + before + ); +}); diff --git a/benchmarks/proof-build/hidden/B-H2/constraints.test.ts b/benchmarks/proof-build/hidden/B-H2/constraints.test.ts new file mode 100644 index 00000000..14e0fd3c --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H2/constraints.test.ts @@ -0,0 +1,32 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// U2: a step goes to the next size on the type scale (8, 9, 10, 11, 12, 14, +// 16, 18, 20, 24, 28, 32, 36, 48, 64, 72, 96); a size between two scale +// sizes goes to the next one in the direction of the step. +function texts(sizes: number[]) { + const e = createEditor(); + const ids = sizes.map((fontSize, i) => + e.create('text', { name: `T${i}`, x: i * 200, fontSize } as never) + ); + e.selection.set(ids); + const sizesNow = () => ids.map((id) => (e.doc.require(id) as any).fontSize); + return { e, sizesNow }; +} + +test('a step up goes to the next size on the type scale', () => { + const { e, sizesNow } = texts([13, 16, 18, 24, 36, 72]); + if (!e.palette.has('Increase font size')) return; + e.palette.run('Increase font size'); + assert.deepEqual(sizesNow(), [14, 18, 20, 28, 48, 96]); + e.palette.run('Increase font size'); + assert.deepEqual(sizesNow(), [16, 20, 24, 32, 64, 104]); +}); + +test('a step down goes to the next size down on the type scale', () => { + const { e, sizesNow } = texts([13, 16, 9, 30, 48, 96]); + if (!e.palette.has('Decrease font size')) return; + e.palette.run('Decrease font size'); + assert.deepEqual(sizesNow(), [12, 14, 8, 28, 36, 72]); +}); diff --git a/benchmarks/proof-build/hidden/B-H2/naive.patch b/benchmarks/proof-build/hidden/B-H2/naive.patch new file mode 100644 index 00000000..09c050f3 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H2/naive.patch @@ -0,0 +1,94 @@ +diff --git a/src/commands/builtin/font-size.ts b/src/commands/builtin/font-size.ts +new file mode 100644 +index 0000000..2d92818 +--- /dev/null ++++ b/src/commands/builtin/font-size.ts +@@ -0,0 +1,40 @@ ++import type { Document } from '../../model/document.ts'; ++import type { TextNode } from '../../model/types.ts'; ++import { isText } from '../../model/types.ts'; ++import { autoHeight } from '../../model/text.ts'; ++import { clamp } from '../../util/math.ts'; ++import type { Command } from '../types.ts'; ++import { command } from '../types.ts'; ++ ++export const MIN_FONT_SIZE = 1; ++export const MAX_FONT_SIZE = 1000; ++ ++/** The size one step up or down from `size`: one pixel. */ ++export function stepFontSize(size: number, up: boolean): number { ++ return up ? size + 1 : size - 1; ++} ++ ++/** The commands that step every text layer in `ids` one size up or down, height included. */ ++export function fontSizeCommands( ++ doc: Document, ++ ids: readonly string[], ++ up: boolean ++): Command[] { ++ const out: Command[] = []; ++ for (const id of ids) { ++ const node = doc.get(id); ++ if (!node || !isText(node)) continue; ++ const fontSize = clamp( ++ stepFontSize(node.fontSize, up), ++ MIN_FONT_SIZE, ++ MAX_FONT_SIZE ++ ); ++ if (fontSize === node.fontSize) continue; ++ const next: TextNode = { ...node, fontSize }; ++ out.push( ++ command('set-prop', { id, key: 'fontSize', value: fontSize }), ++ command('set-prop', { id, key: 'height', value: autoHeight(next) }) ++ ); ++ } ++ return out; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..0d45675 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,8 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { fontSizeCommands } from './builtin/font-size.ts'; ++import { isText } from '../model/types.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -92,6 +94,25 @@ function arrange(title: string, how: Arrange, hint: string): PaletteEntry { + }; + } + ++const hasText = (ctx: EditorContext) => ++ ctx.selection.ids().some((id) => { ++ const node = ctx.doc.get(id); ++ return node !== undefined && isText(node); ++ }); ++ ++function fontSize(title: string, up: boolean): PaletteEntry { ++ return { ++ title, ++ enabled: hasText, ++ run(ctx) { ++ const commands = fontSizeCommands(ctx.doc, ctx.selection.ids(), up); ++ ctx.batch(title, () => { ++ for (const c of commands) ctx.dispatch(c); ++ }); ++ }, ++ }; ++} ++ + export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + { title: 'Undo', hint: 'Mod+Z', run: (ctx) => void ctx.undo() }, + { title: 'Redo', hint: 'Mod+Shift+Z', run: (ctx) => void ctx.redo() }, +@@ -191,6 +212,8 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ fontSize('Increase font size', true), ++ fontSize('Decrease font size', false), + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-H2/reference.patch b/benchmarks/proof-build/hidden/B-H2/reference.patch new file mode 100644 index 00000000..dc4fa407 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H2/reference.patch @@ -0,0 +1,112 @@ +diff --git a/src/commands/builtin/font-size.ts b/src/commands/builtin/font-size.ts +new file mode 100644 +index 0000000..1b906e9 +--- /dev/null ++++ b/src/commands/builtin/font-size.ts +@@ -0,0 +1,58 @@ ++import type { Document } from '../../model/document.ts'; ++import type { TextNode } from '../../model/types.ts'; ++import { isText } from '../../model/types.ts'; ++import { autoHeight } from '../../model/text.ts'; ++import { clamp } from '../../util/math.ts'; ++import type { Command } from '../types.ts'; ++import { command } from '../types.ts'; ++ ++export const MIN_FONT_SIZE = 1; ++export const MAX_FONT_SIZE = 1000; ++ ++/** The type scale a step moves along. */ ++export const TYPE_SCALE: readonly number[] = [ ++ 8, 9, 10, 11, 12, 14, 16, 18, 20, 24, 28, 32, 36, 48, 64, 72, 96, ++]; ++ ++/** ++ * The size one step up or down from `size`: the next size on the type ++ * scale in that direction, so 13 goes up to 14 and down to 12. Below the ++ * scale a step is 1 px, above it 8 px. ++ */ ++export function stepFontSize(size: number, up: boolean): number { ++ const first = TYPE_SCALE[0]; ++ const last = TYPE_SCALE[TYPE_SCALE.length - 1]; ++ if (up) { ++ if (size < first) return Math.min(first, size + 1); ++ if (size >= last) return size + 8; ++ return TYPE_SCALE.find((s) => s > size)!; ++ } ++ if (size <= first) return size - 1; ++ if (size > last) return Math.max(last, size - 8); ++ return [...TYPE_SCALE].reverse().find((s) => s < size)!; ++} ++ ++/** The commands that step every text layer in `ids` one size up or down, height included. */ ++export function fontSizeCommands( ++ doc: Document, ++ ids: readonly string[], ++ up: boolean ++): Command[] { ++ const out: Command[] = []; ++ for (const id of ids) { ++ const node = doc.get(id); ++ if (!node || !isText(node)) continue; ++ const fontSize = clamp( ++ stepFontSize(node.fontSize, up), ++ MIN_FONT_SIZE, ++ MAX_FONT_SIZE ++ ); ++ if (fontSize === node.fontSize) continue; ++ const next: TextNode = { ...node, fontSize }; ++ out.push( ++ command('set-prop', { id, key: 'fontSize', value: fontSize }), ++ command('set-prop', { id, key: 'height', value: autoHeight(next) }) ++ ); ++ } ++ return out; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..0d45675 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,8 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { fontSizeCommands } from './builtin/font-size.ts'; ++import { isText } from '../model/types.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -92,6 +94,25 @@ function arrange(title: string, how: Arrange, hint: string): PaletteEntry { + }; + } + ++const hasText = (ctx: EditorContext) => ++ ctx.selection.ids().some((id) => { ++ const node = ctx.doc.get(id); ++ return node !== undefined && isText(node); ++ }); ++ ++function fontSize(title: string, up: boolean): PaletteEntry { ++ return { ++ title, ++ enabled: hasText, ++ run(ctx) { ++ const commands = fontSizeCommands(ctx.doc, ctx.selection.ids(), up); ++ ctx.batch(title, () => { ++ for (const c of commands) ctx.dispatch(c); ++ }); ++ }, ++ }; ++} ++ + export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + { title: 'Undo', hint: 'Mod+Z', run: (ctx) => void ctx.undo() }, + { title: 'Redo', hint: 'Mod+Shift+Z', run: (ctx) => void ctx.redo() }, +@@ -191,6 +212,8 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ fontSize('Increase font size', true), ++ fontSize('Decrease font size', false), + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-H3/acceptance.test.ts b/benchmarks/proof-build/hidden/B-H3/acceptance.test.ts new file mode 100644 index 00000000..f2bb7621 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H3/acceptance.test.ts @@ -0,0 +1,110 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +function setup() { + const e = createEditor(); + const back = e.create('rect', { + name: 'Backdrop', + x: 0, + y: 0, + width: 600, + height: 400, + }); + const a = e.create('rect', { + name: 'A', + x: 40, + y: 30, + width: 100, + height: 50, + }); + const b = e.create('ellipse', { + name: 'B', + x: 300, + y: 300, + width: 20, + height: 20, + }); + const c = e.create('text', { + name: 'C', + text: 'Label', + x: 200, + y: 120, + width: 120, + height: 20, + } as never); + return { e, back, a, b, c }; +} +type Editor = ReturnType; +const box = (e: Editor, id: string) => { + const { x, y, width, height } = e.doc.require(id); + return { x, y, width, height }; +}; + +test('Frame selection is offered only when something is selected', () => { + const { e, a } = setup(); + assert.ok(!e.palette.list().includes('Frame selection')); + e.selection.set([a]); + assert.ok(e.palette.list().includes('Frame selection')); +}); + +test('the selected layers go into a new frame that fits around them, which is selected', () => { + const { e, a, c } = setup(); + const before = { a: box(e, a), c: box(e, c) }; + e.selection.set([c, a]); + assert.ok(e.palette.run('Frame selection')); + const [frame] = e.selection.ids(); + assert.equal(e.selection.ids().length, 1); + assert.equal(e.doc.require(frame).type, 'frame'); + assert.deepEqual([...e.doc.children(frame)], [a, c]); + assert.deepEqual(box(e, frame), { x: 40, y: 30, width: 280, height: 110 }); + assert.deepEqual({ a: box(e, a), c: box(e, c) }, before); +}); + +test('the frame takes the place of the frontmost framed layer', () => { + const { e, back, a, b, c } = setup(); + e.selection.set([a, c]); + e.palette.run('Frame selection'); + const [frame] = e.selection.ids(); + assert.deepEqual([...e.doc.children(null)], [back, b, frame]); +}); + +test('layers inside a frame are framed where they are', () => { + const e = createEditor(); + const screen = e.create('frame', { name: 'Screen', width: 800, height: 600 }); + const one = e.create( + 'rect', + { name: 'One', x: 10, y: 10, width: 10, height: 10 }, + { parent: screen } + ); + const two = e.create( + 'rect', + { name: 'Two', x: 50, y: 20, width: 10, height: 30 }, + { parent: screen } + ); + const three = e.create( + 'rect', + { name: 'Three', x: 500, y: 500 }, + { parent: screen } + ); + e.selection.set([one, two]); + e.palette.run('Frame selection'); + const [frame] = e.selection.ids(); + assert.equal(e.doc.require(frame).parent, screen); + assert.deepEqual([...e.doc.children(screen)], [frame, three]); + assert.deepEqual(box(e, frame), { x: 10, y: 10, width: 50, height: 40 }); +}); + +test('one Undo takes it all back', () => { + const { e, a, c } = setup(); + const top = [...e.doc.children(null)]; + const size = e.doc.size; + e.selection.set([a, c]); + e.palette.run('Frame selection'); + assert.ok(e.history.undo()); + assert.deepEqual([...e.doc.children(null)], top); + assert.equal(e.doc.size, size); + assert.deepEqual(e.selection.ids(), [a, c]); + assert.ok(e.history.redo()); + assert.equal(e.doc.require(e.selection.ids()[0]).type, 'frame'); +}); diff --git a/benchmarks/proof-build/hidden/B-H3/constraints.test.ts b/benchmarks/proof-build/hidden/B-H3/constraints.test.ts new file mode 100644 index 00000000..33ea785f --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H3/constraints.test.ts @@ -0,0 +1,26 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// U3: a frame made around existing layers has no fill and does not clip, so +// the canvas looks as it did; a frame made from scratch keeps the defaults. +test('a frame made around layers has no fill and does not clip', () => { + const e = createEditor(); + e.create('rect', { name: 'Behind', width: 500, height: 500 }); + const a = e.create('rect', { name: 'A', x: 10, y: 10 }); + const b = e.create('rect', { name: 'B', x: 200, y: 50 }); + if (!e.palette.has('Frame selection')) return; + e.selection.set([a, b]); + e.palette.run('Frame selection'); + const frame = e.doc.require(e.selection.ids()[0]) as any; + if (frame.type !== 'frame') return; + assert.deepEqual(frame.fills, [], 'no fill hides the layers behind'); + assert.equal(frame.clip, false, 'nothing is clipped'); +}); + +test('a frame made from scratch keeps its white fill and clipping', () => { + const e = createEditor(); + const frame = e.doc.require(e.create('frame', { name: 'New' })) as any; + assert.deepEqual(frame.fills, [{ color: '#FFFFFF' }]); + assert.equal(frame.clip, true); +}); diff --git a/benchmarks/proof-build/hidden/B-H3/naive.patch b/benchmarks/proof-build/hidden/B-H3/naive.patch new file mode 100644 index 00000000..dc50bbb8 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H3/naive.patch @@ -0,0 +1,84 @@ +diff --git a/src/commands/builtin/group.ts b/src/commands/builtin/group.ts +index b977927..1cacf79 100644 +--- a/src/commands/builtin/group.ts ++++ b/src/commands/builtin/group.ts +@@ -1,4 +1,5 @@ +-import { createGroup } from '../../model/factory.ts'; ++import { createFrame, createGroup } from '../../model/factory.ts'; ++import { selectionBounds } from '../../selection/bounds.ts'; + import { + commonParent, + indexInParent, +@@ -50,3 +51,47 @@ export function groupNodes( + }); + return groupId; + } ++ ++/** ++ * Puts `ids` into a new frame that fits tightly around them, placed where ++ * the frontmost of them was, then selects the frame. The layers keep their ++ * places on the canvas. Returns the frame's id, or null when nothing can be ++ * framed. ++ */ ++export function frameNodes( ++ ctx: EditorContext, ++ ids: readonly string[] ++): string | null { ++ const { doc } = ctx; ++ const members = sortByPaintOrder( ++ doc, ++ topmostOnly( ++ doc, ++ ids.filter((id) => doc.has(id)) ++ ) ++ ); ++ if (!members.length) return null; ++ const parent = commonParent(doc, members); ++ const front = members[members.length - 1]; ++ let anchor = front; ++ while (doc.require(anchor).parent !== parent) ++ anchor = doc.require(anchor).parent!; ++ const bounds = selectionBounds(doc, members)!; ++ const frame = createFrame(ctx.ids, { ...bounds }); ++ const frameId = frame.id; ++ ctx.batch('Frame selection', () => { ++ ctx.dispatch( ++ command('insert', { ++ nodes: [frame] as unknown as Json[], ++ parent, ++ index: indexInParent(doc, anchor) + 1, ++ }) ++ ); ++ members.forEach((id, i) => { ++ ctx.dispatch(command('reorder', { id, parent: frameId, index: i }), { ++ select: i === members.length - 1 ? [frameId] : undefined, ++ }); ++ }); ++ }); ++ return frameId; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..e752622 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,7 +1,7 @@ + import { fail } from '../util/assert.ts'; + import type { EditorContext } from './types.ts'; + import { command } from './types.ts'; +-import { groupNodes } from './builtin/group.ts'; ++import { frameNodes, groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; + import type { Arrange } from './builtin/reorder.ts'; + import { visibilityItems } from './builtin/visibility.ts'; +@@ -130,6 +130,11 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + enabled: hasSelection, + run: (ctx) => void groupNodes(ctx, ctx.selection.ids()), + }, ++ { ++ title: 'Frame selection', ++ enabled: hasSelection, ++ run: (ctx) => void frameNodes(ctx, ctx.selection.ids()), ++ }, + arrange('Bring forward', 'forward', 'Mod+]'), + arrange('Send backward', 'backward', 'Mod+['), + arrange('Bring to front', 'front', 'Mod+Shift+]'), diff --git a/benchmarks/proof-build/hidden/B-H3/reference.patch b/benchmarks/proof-build/hidden/B-H3/reference.patch new file mode 100644 index 00000000..25ed6366 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H3/reference.patch @@ -0,0 +1,84 @@ +diff --git a/src/commands/builtin/group.ts b/src/commands/builtin/group.ts +index b977927..20e4744 100644 +--- a/src/commands/builtin/group.ts ++++ b/src/commands/builtin/group.ts +@@ -1,4 +1,5 @@ +-import { createGroup } from '../../model/factory.ts'; ++import { createFrame, createGroup } from '../../model/factory.ts'; ++import { selectionBounds } from '../../selection/bounds.ts'; + import { + commonParent, + indexInParent, +@@ -50,3 +51,47 @@ export function groupNodes( + }); + return groupId; + } ++ ++/** ++ * Puts `ids` into a new frame that fits tightly around them, placed where ++ * the frontmost of them was, then selects the frame. The layers keep their ++ * places on the canvas. Returns the frame's id, or null when nothing can be ++ * framed. ++ */ ++export function frameNodes( ++ ctx: EditorContext, ++ ids: readonly string[] ++): string | null { ++ const { doc } = ctx; ++ const members = sortByPaintOrder( ++ doc, ++ topmostOnly( ++ doc, ++ ids.filter((id) => doc.has(id)) ++ ) ++ ); ++ if (!members.length) return null; ++ const parent = commonParent(doc, members); ++ const front = members[members.length - 1]; ++ let anchor = front; ++ while (doc.require(anchor).parent !== parent) ++ anchor = doc.require(anchor).parent!; ++ const bounds = selectionBounds(doc, members)!; ++ const frame = createFrame(ctx.ids, { ...bounds, fills: [], clip: false }); ++ const frameId = frame.id; ++ ctx.batch('Frame selection', () => { ++ ctx.dispatch( ++ command('insert', { ++ nodes: [frame] as unknown as Json[], ++ parent, ++ index: indexInParent(doc, anchor) + 1, ++ }) ++ ); ++ members.forEach((id, i) => { ++ ctx.dispatch(command('reorder', { id, parent: frameId, index: i }), { ++ select: i === members.length - 1 ? [frameId] : undefined, ++ }); ++ }); ++ }); ++ return frameId; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..e752622 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -1,7 +1,7 @@ + import { fail } from '../util/assert.ts'; + import type { EditorContext } from './types.ts'; + import { command } from './types.ts'; +-import { groupNodes } from './builtin/group.ts'; ++import { frameNodes, groupNodes } from './builtin/group.ts'; + import { arrangeCommands } from './builtin/reorder.ts'; + import type { Arrange } from './builtin/reorder.ts'; + import { visibilityItems } from './builtin/visibility.ts'; +@@ -130,6 +130,11 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + enabled: hasSelection, + run: (ctx) => void groupNodes(ctx, ctx.selection.ids()), + }, ++ { ++ title: 'Frame selection', ++ enabled: hasSelection, ++ run: (ctx) => void frameNodes(ctx, ctx.selection.ids()), ++ }, + arrange('Bring forward', 'forward', 'Mod+]'), + arrange('Send backward', 'backward', 'Mod+['), + arrange('Bring to front', 'front', 'Mod+Shift+]'), diff --git a/benchmarks/proof-build/hidden/B-H4/acceptance.test.ts b/benchmarks/proof-build/hidden/B-H4/acceptance.test.ts new file mode 100644 index 00000000..129b1f2e --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H4/acceptance.test.ts @@ -0,0 +1,96 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +const RED = '#FF0000'; +const GREEN = '#00AA00'; +const BLUE = '#0000FF'; + +function setup() { + const e = createEditor(); + const screen = e.create('frame', { name: 'Screen', width: 800, height: 600 }); + const icons = e.create('group', { name: 'Icons' }, { parent: screen }); + const dot = e.create( + 'ellipse', + { name: 'Dot', fills: [{ color: RED }] } as never, + { parent: icons } + ); + const ring = e.create( + 'ellipse', + { + name: 'Ring', + x: 50, + fills: [{ color: BLUE }], + strokes: [{ color: RED, width: 2, align: 'inside' }], + } as never, + { parent: icons } + ); + const label = e.create( + 'text', + { name: 'Label', text: 'Sale', y: 200, fills: [{ color: RED }] } as never, + { parent: screen } + ); + const loose = e.create('rect', { + name: 'Loose', + x: 900, + fills: [{ color: RED }], + } as never); + const other = e.create('rect', { + name: 'Other', + x: 1100, + fills: [{ color: BLUE }], + } as never); + return { e, screen, dot, ring, label, loose, other }; +} +type Editor = ReturnType; +const fill = (e: Editor, id: string) => + (e.doc.require(id) as any).fills[0]?.color; +const stroke = (e: Editor, id: string) => + (e.doc.require(id) as any).strokes[0]?.color; + +test('every fill and stroke in the first colour gets the second, whatever is selected', () => { + const { e, screen, dot, ring, label, loose, other } = setup(); + assert.ok(e.palette.run('Replace colour', '#ff0000 #00aa00')); + assert.equal(fill(e, dot), GREEN); + assert.equal(stroke(e, ring), GREEN); + assert.equal(fill(e, ring), BLUE); + assert.equal(fill(e, label), GREEN); + assert.equal(fill(e, loose), GREEN); + assert.equal(fill(e, other), BLUE); + assert.equal(fill(e, screen), '#FFFFFF'); +}); + +test('the selection does not limit it', () => { + const { e, dot, loose, other } = setup(); + e.selection.set([other]); + e.palette.run('Replace colour', `${RED} ${GREEN}`); + assert.equal(fill(e, dot), GREEN); + assert.equal(fill(e, loose), GREEN); + assert.equal(fill(e, other), BLUE); +}); + +test('input it cannot read changes nothing', () => { + const { e } = setup(); + const saved = e.files.save(); + for (const input of [ + 'red green', + '#FF0000', + '', + 'FF0000 to 00AA00', + '#GG0000 #00AA00', + ]) { + e.palette.run('Replace colour', input); + assert.equal(e.files.save(), saved, `input "${input}"`); + } + assert.equal(e.history.size, 0); +}); + +test('one Undo puts the old colour back everywhere', () => { + const { e } = setup(); + const saved = e.files.save(); + e.palette.run('Replace colour', `${RED} ${GREEN}`); + assert.notEqual(e.files.save(), saved); + assert.equal(e.history.size, 1); + assert.ok(e.history.undo()); + assert.equal(e.files.save(), saved); +}); diff --git a/benchmarks/proof-build/hidden/B-H4/constraints.test.ts b/benchmarks/proof-build/hidden/B-H4/constraints.test.ts new file mode 100644 index 00000000..f72fcebd --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H4/constraints.test.ts @@ -0,0 +1,46 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// U4: an edit that sweeps the whole document leaves locked layers, and +// everything inside a locked group or frame, alone, and changes hidden +// layers like any other. +const RED = '#FF0000'; +const GREEN = '#00AA00'; + +test('locked layers keep their colour; hidden ones change', () => { + const e = createEditor(); + const red = (name: string, extra = {}, parent?: string) => + e.create( + 'rect', + { name, fills: [{ color: RED }], ...extra } as never, + parent ? { parent } : {} + ); + const plain = red('Plain'); + const lockedRect = red('Locked logo', { locked: true }); + const lockedGroup = e.create('group', { + name: 'Approved', + locked: true, + } as never); + const inLockedGroup = red('Mark', {}, lockedGroup); + const lockedFrame = e.create('frame', { + name: 'Signed off', + locked: true, + } as never); + const inLockedFrame = red('Button', {}, lockedFrame); + const hiddenRect = red('Hover state', { visible: false }); + const hiddenGroup = e.create('group', { + name: 'States', + visible: false, + } as never); + const inHiddenGroup = red('Pressed', {}, hiddenGroup); + if (!e.palette.has('Replace colour')) return; + e.palette.run('Replace colour', `${RED} ${GREEN}`); + const fill = (id: string) => (e.doc.require(id) as any).fills[0].color; + if (fill(plain) !== GREEN) return; + assert.equal(fill(lockedRect), RED, 'a locked layer'); + assert.equal(fill(inLockedGroup), RED, 'inside a locked group'); + assert.equal(fill(inLockedFrame), RED, 'inside a locked frame'); + assert.equal(fill(hiddenRect), GREEN, 'a hidden layer'); + assert.equal(fill(inHiddenGroup), GREEN, 'inside a hidden group'); +}); diff --git a/benchmarks/proof-build/hidden/B-H4/naive.patch b/benchmarks/proof-build/hidden/B-H4/naive.patch new file mode 100644 index 00000000..50cd31bb --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H4/naive.patch @@ -0,0 +1,88 @@ +diff --git a/src/commands/builtin/recolour.ts b/src/commands/builtin/recolour.ts +new file mode 100644 +index 0000000..6ca229d +--- /dev/null ++++ b/src/commands/builtin/recolour.ts +@@ -0,0 +1,52 @@ ++import type { Document } from '../../model/document.ts'; ++import type { Paint, Stroke } from '../../model/types.ts'; ++import { isStyled } from '../../model/types.ts'; ++import { walk } from '../../model/query.ts'; ++import { parseColor } from '../../model/style.ts'; ++import type { Command, Json } from '../types.ts'; ++import { command } from '../types.ts'; ++ ++/** Reads "#FF0000 #00AA00": the colour to find and the colour to use. */ ++export function parseColourPair( ++ input: string | undefined ++): { from: string; to: string } | null { ++ const parts = (input ?? '').trim().split(/\s+/); ++ if (parts.length !== 2) return null; ++ const from = parseColor(parts[0]); ++ const to = parseColor(parts[1]); ++ return from && to ? { from, to } : null; ++} ++ ++/** ++ * The set-prop commands that give every fill and stroke in `from` the ++ * colour `to`, across the whole document, wherever the layer sits. ++ */ ++export function recolourCommands( ++ doc: Document, ++ from: string, ++ to: string ++): Command[] { ++ const out: Command[] = []; ++ walk(doc, (node) => { ++ if (!isStyled(node)) return; ++ const swap = (paints: T[]): T[] => ++ paints.map((p) => (p.color === from ? { ...p, color: to } : p)); ++ if (node.fills.some((p) => p.color === from)) ++ out.push( ++ command('set-prop', { ++ id: node.id, ++ key: 'fills', ++ value: swap(node.fills) as unknown as Json[], ++ }) ++ ); ++ if (node.strokes.some((s) => s.color === from)) ++ out.push( ++ command('set-prop', { ++ id: node.id, ++ key: 'strokes', ++ value: swap(node.strokes) as unknown as Json[], ++ }) ++ ); ++ }); ++ return out; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..ed76bad 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,7 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { parseColourPair, recolourCommands } from './builtin/recolour.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -191,6 +192,17 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ { ++ title: 'Replace colour', ++ run(ctx, input) { ++ const pair = parseColourPair(input); ++ if (!pair || pair.from === pair.to) return; ++ const commands = recolourCommands(ctx.doc, pair.from, pair.to); ++ ctx.batch('Replace colour', () => { ++ for (const c of commands) ctx.dispatch(c); ++ }); ++ }, ++ }, + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-H4/reference.patch b/benchmarks/proof-build/hidden/B-H4/reference.patch new file mode 100644 index 00000000..135cfc9f --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H4/reference.patch @@ -0,0 +1,94 @@ +diff --git a/src/commands/builtin/recolour.ts b/src/commands/builtin/recolour.ts +new file mode 100644 +index 0000000..8402eae +--- /dev/null ++++ b/src/commands/builtin/recolour.ts +@@ -0,0 +1,58 @@ ++import type { Document } from '../../model/document.ts'; ++import type { Paint, Stroke } from '../../model/types.ts'; ++import { isStyled } from '../../model/types.ts'; ++import { walk } from '../../model/query.ts'; ++import { ancestors } from '../../model/tree.ts'; ++import { parseColor } from '../../model/style.ts'; ++import type { Command, Json } from '../types.ts'; ++import { command } from '../types.ts'; ++ ++/** Reads "#FF0000 #00AA00": the colour to find and the colour to use. */ ++export function parseColourPair( ++ input: string | undefined ++): { from: string; to: string } | null { ++ const parts = (input ?? '').trim().split(/\s+/); ++ if (parts.length !== 2) return null; ++ const from = parseColor(parts[0]); ++ const to = parseColor(parts[1]); ++ return from && to ? { from, to } : null; ++} ++ ++/** ++ * The set-prop commands that give every fill and stroke in `from` the ++ * colour `to`, across the whole document. Locked layers, and everything ++ * inside a locked container, are left alone; hidden layers change like any ++ * other. ++ */ ++export function recolourCommands( ++ doc: Document, ++ from: string, ++ to: string ++): Command[] { ++ const out: Command[] = []; ++ walk(doc, (node) => { ++ if (!isStyled(node)) return; ++ // Locked layers, and everything inside a locked group or frame, keep ++ // their colours. ++ if (node.locked || ancestors(doc, node.id).some((a) => a.locked)) return; ++ const swap = (paints: T[]): T[] => ++ paints.map((p) => (p.color === from ? { ...p, color: to } : p)); ++ if (node.fills.some((p) => p.color === from)) ++ out.push( ++ command('set-prop', { ++ id: node.id, ++ key: 'fills', ++ value: swap(node.fills) as unknown as Json[], ++ }) ++ ); ++ if (node.strokes.some((s) => s.color === from)) ++ out.push( ++ command('set-prop', { ++ id: node.id, ++ key: 'strokes', ++ value: swap(node.strokes) as unknown as Json[], ++ }) ++ ); ++ }); ++ return out; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..ed76bad 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,7 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { parseColourPair, recolourCommands } from './builtin/recolour.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -191,6 +192,17 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ); + }, + }, ++ { ++ title: 'Replace colour', ++ run(ctx, input) { ++ const pair = parseColourPair(input); ++ if (!pair || pair.from === pair.to) return; ++ const commands = recolourCommands(ctx.doc, pair.from, pair.to); ++ ctx.batch('Replace colour', () => { ++ for (const c of commands) ctx.dispatch(c); ++ }); ++ }, ++ }, + { + title: 'Rename layer', + enabled: (ctx) => ctx.selection.size === 1, diff --git a/benchmarks/proof-build/hidden/B-H5/acceptance.test.ts b/benchmarks/proof-build/hidden/B-H5/acceptance.test.ts new file mode 100644 index 00000000..6dc37bc8 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H5/acceptance.test.ts @@ -0,0 +1,82 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +const RED = [{ color: '#FF0000' }]; +const BLUE = [{ color: '#0000FF' }]; + +/** Layers straight on the page, one of them in a group; no frames. */ +function page() { + const e = createEditor(); + const a = e.create('rect', { name: 'A', fills: RED } as never); + const g = e.create('group', { name: 'G' }); + const b = e.create('ellipse', { name: 'B', x: 100, fills: RED } as never, { + parent: g, + }); + const c = e.create('rect', { name: 'C', x: 200, fills: BLUE } as never); + const t = e.create('text', { + name: 'T', + x: 300, + text: 'Hi', + fills: RED, + } as never); + const line = e.create('line', { name: 'Rule', y: 200, width: 300 }); + return { e, a, g, b, c, t, line }; +} +type Editor = ReturnType; +const picked = (e: Editor) => [...e.selection.ids()].sort(); + +test('it selects the layers with the same fill colour, groups included', () => { + const { e, a, b, t } = page(); + e.selection.set([a]); + assert.ok(e.palette.run('Select same fill')); + assert.deepEqual(picked(e), [a, b, t].sort()); +}); + +test('it starts from whichever layer is selected', () => { + const { e, c } = page(); + e.selection.set([c]); + e.palette.run('Select same fill'); + assert.deepEqual(picked(e), [c]); +}); + +test('inside one frame it finds the matches at any depth', () => { + const e = createEditor(); + const screen = e.create('frame', { name: 'Screen', width: 800, height: 600 }); + const card = e.create('frame', { name: 'Card', fills: BLUE } as never, { + parent: screen, + }); + const x = e.create('rect', { name: 'X', fills: RED } as never, { + parent: screen, + }); + const g = e.create('group', { name: 'Row' }, { parent: card }); + const y = e.create('rect', { name: 'Y', x: 50, fills: RED } as never, { + parent: g, + }); + e.create('rect', { name: 'Z', x: 90, fills: BLUE } as never, { + parent: card, + }); + e.selection.set([y]); + e.palette.run('Select same fill'); + assert.deepEqual(picked(e), [x, y].sort()); +}); + +test('it is offered only when exactly one layer with a fill is selected', () => { + const { e, a, c, g, line } = page(); + const offered = (ids: string[]) => { + e.selection.set(ids); + return e.palette.list().includes('Select same fill'); + }; + assert.equal(offered([]), false); + assert.equal(offered([a, c]), false); + assert.equal(offered([g]), false); + assert.equal(offered([line]), false); + assert.equal(offered([a]), true); +}); + +test('selecting is not an edit', () => { + const { e, a } = page(); + e.selection.set([a]); + e.palette.run('Select same fill'); + assert.equal(e.history.size, 0); +}); diff --git a/benchmarks/proof-build/hidden/B-H5/constraints.test.ts b/benchmarks/proof-build/hidden/B-H5/constraints.test.ts new file mode 100644 index 00000000..c51d64c3 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H5/constraints.test.ts @@ -0,0 +1,59 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// U5: picking similar layers stays inside the top-level frame that holds +// the starting layer, at any depth; a layer outside every frame looks only +// at the other layers outside frames. +const RED = [{ color: '#FF0000' }]; + +function screens() { + const e = createEditor(); + const red = (name: string, parent?: string, x = 0) => + e.create( + 'rect', + { name, x, fills: RED } as never, + parent ? { parent } : {} + ); + const home = e.create('frame', { name: 'Home', width: 400, height: 800 }); + const homeButton = red('Home button', home); + const homeCard = e.create( + 'frame', + { name: 'Card', fills: [{ color: '#FFFFFF' }] } as never, + { parent: home } + ); + const homeRow = e.create('group', { name: 'Row' }, { parent: homeCard }); + const homeBadge = red('Badge', homeRow, 20); + const settings = e.create('frame', { + name: 'Settings', + x: 500, + width: 400, + height: 800, + }); + red('Settings button', settings, 520); + const looseGroup = e.create('group', { name: 'Notes' }); + const loose = red('Sticker', undefined, 1000); + const looseInGroup = red('Arrow', looseGroup, 1100); + return { e, homeButton, homeBadge, loose, looseInGroup }; +} + +test('the search stays inside the top-level frame of the selected layer', () => { + const { e, homeButton, homeBadge } = screens(); + if (!e.palette.has('Select same fill')) return; + e.selection.set([homeButton]); + e.palette.run('Select same fill'); + if (e.selection.ids().length < 2) return; + assert.deepEqual( + [...e.selection.ids()].sort(), + [homeBadge, homeButton].sort() + ); +}); + +test('a layer outside every frame finds only the others outside frames', () => { + const { e, loose, looseInGroup } = screens(); + if (!e.palette.has('Select same fill')) return; + e.selection.set([loose]); + e.palette.run('Select same fill'); + if (e.selection.ids().length < 2) return; + assert.deepEqual([...e.selection.ids()].sort(), [loose, looseInGroup].sort()); +}); diff --git a/benchmarks/proof-build/hidden/B-H5/naive.patch b/benchmarks/proof-build/hidden/B-H5/naive.patch new file mode 100644 index 00000000..43a9a1e6 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H5/naive.patch @@ -0,0 +1,54 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..061721d 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,8 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { sameFill } from '../selection/same.ts'; ++import { firstFill } from '../model/style.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -108,6 +110,18 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ctx.selection.set(ids); + }, + }, ++ { ++ title: 'Select same fill', ++ enabled(ctx) { ++ if (ctx.selection.size !== 1) return false; ++ const node = ctx.doc.get(ctx.selection.ids()[0]); ++ return node !== undefined && firstFill(node) !== null; ++ }, ++ run(ctx) { ++ const ids = sameFill(ctx.doc, ctx.selection.ids()[0]); ++ if (ids.length) ctx.selection.set(ids); ++ }, ++ }, + { + title: 'Deselect all', + hint: 'Escape', +diff --git a/src/selection/same.ts b/src/selection/same.ts +new file mode 100644 +index 0000000..ec1abbb +--- /dev/null ++++ b/src/selection/same.ts +@@ -0,0 +1,16 @@ ++import type { Document } from '../model/document.ts'; ++import { firstFill } from '../model/style.ts'; ++import { walk } from '../model/query.ts'; ++import { isInteractive } from './rules.ts'; ++ ++/** `id` and every other layer whose fill is the same colour as its fill. */ ++export function sameFill(doc: Document, id: string): string[] { ++ const color = firstFill(doc.require(id))?.color; ++ if (!color) return []; ++ const matches: string[] = []; ++ walk(doc, (node) => { ++ if (node.id === id || firstFill(node)?.color !== color) return; ++ if (isInteractive(doc, node.id)) matches.push(node.id); ++ }); ++ return [id, ...matches]; ++} diff --git a/benchmarks/proof-build/hidden/B-H5/reference.patch b/benchmarks/proof-build/hidden/B-H5/reference.patch new file mode 100644 index 00000000..0de9f288 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H5/reference.patch @@ -0,0 +1,73 @@ +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..061721d 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,8 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { sameFill } from '../selection/same.ts'; ++import { firstFill } from '../model/style.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -108,6 +110,18 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + ctx.selection.set(ids); + }, + }, ++ { ++ title: 'Select same fill', ++ enabled(ctx) { ++ if (ctx.selection.size !== 1) return false; ++ const node = ctx.doc.get(ctx.selection.ids()[0]); ++ return node !== undefined && firstFill(node) !== null; ++ }, ++ run(ctx) { ++ const ids = sameFill(ctx.doc, ctx.selection.ids()[0]); ++ if (ids.length) ctx.selection.set(ids); ++ }, ++ }, + { + title: 'Deselect all', + hint: 'Escape', +diff --git a/src/selection/same.ts b/src/selection/same.ts +new file mode 100644 +index 0000000..77b8123 +--- /dev/null ++++ b/src/selection/same.ts +@@ -0,0 +1,35 @@ ++import type { Document } from '../model/document.ts'; ++import type { SceneNode } from '../model/types.ts'; ++import { firstFill } from '../model/style.ts'; ++import { walk } from '../model/query.ts'; ++import { ancestors, descendants } from '../model/tree.ts'; ++import { isInteractive } from './rules.ts'; ++ ++/** The top-level frame that holds `id`, or null when it sits outside every frame. */ ++function topFrame(doc: Document, id: string): string | null { ++ const frames = ancestors(doc, id).filter((a) => a.type === 'frame'); ++ return frames.length ? frames[frames.length - 1].id : null; ++} ++ ++/** ++ * `id` and the layers whose fill is the same colour as its fill. The search ++ * stays inside the top-level frame that holds `id`, at any depth; a layer ++ * outside every frame looks only at the other layers outside frames. ++ */ ++export function sameFill(doc: Document, id: string): string[] { ++ const color = firstFill(doc.require(id))?.color; ++ if (!color) return []; ++ const frame = topFrame(doc, id); ++ const pool: SceneNode[] = []; ++ if (frame !== null) pool.push(...descendants(doc, frame)); ++ else ++ walk(doc, (node) => { ++ if (node.type === 'frame') return false; ++ pool.push(node); ++ }); ++ const matches = pool ++ .filter((node) => node.id !== id && firstFill(node)?.color === color) ++ .filter((node) => isInteractive(doc, node.id)) ++ .map((node) => node.id); ++ return [id, ...matches]; ++} diff --git a/benchmarks/proof-build/hidden/B-H6/acceptance.test.ts b/benchmarks/proof-build/hidden/B-H6/acceptance.test.ts new file mode 100644 index 00000000..1efb9894 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H6/acceptance.test.ts @@ -0,0 +1,118 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +type Editor = ReturnType; +const box = (e: Editor, id: string) => { + const { x, y, width, height } = e.doc.require(id); + return { x, y, width, height }; +}; + +function assertWhole(e: Editor, id: string, was: ReturnType) { + const now = box(e, id); + for (const key of ['x', 'y', 'width', 'height'] as const) { + assert.ok(Number.isInteger(now[key]), `${key} ${now[key]} is whole`); + assert.ok( + Math.abs(now[key] - was[key]) <= 1, + `${key} moved from ${was[key]} to ${now[key]}` + ); + } +} + +test('the selected layers end up on whole pixels', () => { + const e = createEditor(); + const a = e.create('rect', { + name: 'A', + x: 10.3, + y: 4.6, + width: 20.2, + height: 9.7, + }); + const b = e.create('ellipse', { + name: 'B', + x: 99.5, + y: 0.25, + width: 30.4, + height: 30.4, + }); + const before = { a: box(e, a), b: box(e, b) }; + e.selection.set([a, b]); + assert.ok(e.palette.run('Round to pixels')); + assertWhole(e, a, before.a); + assertWhole(e, b, before.b); + e.selection.set([a]); + for (const label of ['X', 'Y', 'W', 'H']) + assert.match(e.panel.value(label)!, /^-?\d+$/, label); +}); + +test('layers inside selected groups and frames are rounded too; others are not', () => { + const e = createEditor(); + const frame = e.create('frame', { + name: 'Icon', + x: 0.4, + y: 0.6, + width: 47.8, + height: 48.3, + }); + const inFrame = e.create( + 'rect', + { name: 'Body', x: 4.2, y: 4.7, width: 20.6, height: 11.1 }, + { parent: frame } + ); + const group = e.create('group', { name: 'Marks' }); + const inGroup = e.create( + 'rect', + { name: 'Tick', x: 100.7, y: 3.3, width: 5.5, height: 2.2 }, + { parent: group } + ); + const untouched = e.create('rect', { + name: 'Other', + x: 300.5, + y: 1.5, + width: 2.5, + height: 2.5, + }); + const before = { + frame: box(e, frame), + inFrame: box(e, inFrame), + inGroup: box(e, inGroup), + untouched: box(e, untouched), + }; + e.selection.set([frame, group]); + e.palette.run('Round to pixels'); + assertWhole(e, frame, before.frame); + assertWhole(e, inFrame, before.inFrame); + assertWhole(e, inGroup, before.inGroup); + assertWhole(e, group, before.inGroup); + assert.deepEqual(box(e, untouched), before.untouched); +}); + +test('one Undo puts them all back exactly', () => { + const e = createEditor(); + const ids = [0.1, 1.2, 2.3].map((f, i) => + e.create('rect', { + name: `R${i}`, + x: i * 50 + f, + y: f, + width: 20 + f, + height: 10 + f, + }) + ); + const before = ids.map((id) => box(e, id)); + e.selection.set(ids); + e.palette.run('Round to pixels'); + assert.equal(e.history.size, 1); + assert.ok(e.history.undo()); + assert.deepEqual( + ids.map((id) => box(e, id)), + before + ); +}); + +test('it is offered when something is selected', () => { + const e = createEditor(); + const a = e.create('rect', { name: 'A' }); + assert.ok(!e.palette.list().includes('Round to pixels')); + e.selection.set([a]); + assert.ok(e.palette.list().includes('Round to pixels')); +}); diff --git a/benchmarks/proof-build/hidden/B-H6/constraints.test.ts b/benchmarks/proof-build/hidden/B-H6/constraints.test.ts new file mode 100644 index 00000000..d55adcf0 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H6/constraints.test.ts @@ -0,0 +1,52 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createEditor } from '../../src/editor.ts'; + +// U6: a whole-pixel fit rounds each edge to the nearest whole pixel and +// takes the size from the rounded edges, so shapes that touched still touch. +test('shapes that touched still touch after rounding', () => { + const e = createEditor(); + const left = e.create('rect', { + name: 'Left', + x: 0.4, + y: 0, + width: 10.1, + height: 8, + }); + const right = e.create('rect', { + name: 'Right', + x: 10.5, + y: 0, + width: 10.2, + height: 8, + }); + const top = e.create('rect', { + name: 'Top', + x: 40, + y: 0.4, + width: 8, + height: 10.1, + }); + const bottom = e.create('rect', { + name: 'Bottom', + x: 40, + y: 10.5, + width: 8, + height: 9.6, + }); + if (!e.palette.has('Round to pixels')) return; + e.selection.set([left, right, top, bottom]); + e.palette.run('Round to pixels'); + const n = (id: string) => e.doc.require(id); + if (!Number.isInteger(n(left).x)) return; + assert.equal(n(left).x + n(left).width, n(right).x, 'left meets right'); + assert.equal(n(top).y + n(top).height, n(bottom).y, 'top meets bottom'); + assert.deepEqual( + [n(left).x, n(left).width, n(right).x, n(right).width], + [0, 11, 11, 10] + ); + assert.deepEqual( + [n(top).y, n(top).height, n(bottom).y, n(bottom).height], + [0, 11, 11, 9] + ); +}); diff --git a/benchmarks/proof-build/hidden/B-H6/naive.patch b/benchmarks/proof-build/hidden/B-H6/naive.patch new file mode 100644 index 00000000..77255e89 --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H6/naive.patch @@ -0,0 +1,82 @@ +diff --git a/src/commands/builtin/pixels.ts b/src/commands/builtin/pixels.ts +new file mode 100644 +index 0000000..bc84428 +--- /dev/null ++++ b/src/commands/builtin/pixels.ts +@@ -0,0 +1,49 @@ ++import type { Document } from '../../model/document.ts'; ++import type { Rect } from '../../model/geometry.ts'; ++import { rectOf } from '../../model/geometry.ts'; ++import { descendants } from '../../model/tree.ts'; ++import type { Command, Json } from '../types.ts'; ++import { command } from '../types.ts'; ++import type { GeometryItem } from './resize.ts'; ++ ++/** A rect with x, y, width and height each rounded to whole pixels. */ ++export function roundRect(r: Rect): Rect { ++ return { ++ x: Math.round(r.x), ++ y: Math.round(r.y), ++ width: r.width === 0 ? 0 : Math.max(1, Math.round(r.width)), ++ height: r.height === 0 ? 0 : Math.max(1, Math.round(r.height)), ++ }; ++} ++ ++/** ++ * One resize command that puts every layer in `ids`, and every layer ++ * inside them, on whole pixels. Groups follow their children. Returns null ++ * when everything is on whole pixels already. ++ */ ++export function roundToPixelsCommand( ++ doc: Document, ++ ids: readonly string[] ++): Command | null { ++ const seen = new Set(); ++ const items: GeometryItem[] = []; ++ for (const id of ids) { ++ if (!doc.has(id)) continue; ++ for (const node of [doc.require(id), ...descendants(doc, id)]) { ++ if (node.type === 'group' || seen.has(node.id)) continue; ++ seen.add(node.id); ++ const from = rectOf(node); ++ const to = roundRect(from); ++ if ( ++ to.x !== from.x || ++ to.y !== from.y || ++ to.width !== from.width || ++ to.height !== from.height ++ ) ++ items.push({ id: node.id, ...to }); ++ } ++ } ++ return items.length ++ ? command('resize', { items: items as unknown as Json[] }) ++ : null; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..7133b67 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,7 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { roundToPixelsCommand } from './builtin/pixels.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -142,6 +143,14 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + align('Align bottom', 'bottom'), + distribute('Distribute horizontally', 'horizontal'), + distribute('Distribute vertically', 'vertical'), ++ { ++ title: 'Round to pixels', ++ enabled: hasSelection, ++ run(ctx) { ++ const c = roundToPixelsCommand(ctx.doc, ctx.selection.ids()); ++ if (c) ctx.dispatch(c, { label: 'Round to pixels' }); ++ }, ++ }, + { + title: 'Hide', + enabled: hasSelection, diff --git a/benchmarks/proof-build/hidden/B-H6/reference.patch b/benchmarks/proof-build/hidden/B-H6/reference.patch new file mode 100644 index 00000000..e01b87df --- /dev/null +++ b/benchmarks/proof-build/hidden/B-H6/reference.patch @@ -0,0 +1,91 @@ +diff --git a/src/commands/builtin/pixels.ts b/src/commands/builtin/pixels.ts +new file mode 100644 +index 0000000..f192260 +--- /dev/null ++++ b/src/commands/builtin/pixels.ts +@@ -0,0 +1,58 @@ ++import type { Document } from '../../model/document.ts'; ++import type { Rect } from '../../model/geometry.ts'; ++import { rectOf } from '../../model/geometry.ts'; ++import { descendants } from '../../model/tree.ts'; ++import type { Command, Json } from '../types.ts'; ++import { command } from '../types.ts'; ++import type { GeometryItem } from './resize.ts'; ++ ++/** ++ * A rect with each edge (left, top, right, bottom) on the nearest whole ++ * pixel, width and height taken from the rounded edges, so rects that ++ * touched still touch. A size that would drop below 1 px is 1 px; a zero ++ * size (a line's height) stays zero. ++ */ ++export function roundRect(r: Rect): Rect { ++ const left = Math.round(r.x); ++ const top = Math.round(r.y); ++ const size = (from: number, to: number, was: number) => ++ was === 0 ? 0 : Math.max(1, to - from); ++ return { ++ x: left, ++ y: top, ++ width: size(left, Math.round(r.x + r.width), r.width), ++ height: size(top, Math.round(r.y + r.height), r.height), ++ }; ++} ++ ++/** ++ * One resize command that puts every layer in `ids`, and every layer ++ * inside them, on whole pixels. Groups follow their children. Returns null ++ * when everything is on whole pixels already. ++ */ ++export function roundToPixelsCommand( ++ doc: Document, ++ ids: readonly string[] ++): Command | null { ++ const seen = new Set(); ++ const items: GeometryItem[] = []; ++ for (const id of ids) { ++ if (!doc.has(id)) continue; ++ for (const node of [doc.require(id), ...descendants(doc, id)]) { ++ if (node.type === 'group' || seen.has(node.id)) continue; ++ seen.add(node.id); ++ const from = rectOf(node); ++ const to = roundRect(from); ++ if ( ++ to.x !== from.x || ++ to.y !== from.y || ++ to.width !== from.width || ++ to.height !== from.height ++ ) ++ items.push({ id: node.id, ...to }); ++ } ++ } ++ return items.length ++ ? command('resize', { items: items as unknown as Json[] }) ++ : null; ++} +diff --git a/src/commands/palette.ts b/src/commands/palette.ts +index 874d40e..7133b67 100644 +--- a/src/commands/palette.ts ++++ b/src/commands/palette.ts +@@ -10,6 +10,7 @@ import { alignCommand, distributeCommand } from './builtin/align.ts'; + import type { Alignment, Axis } from './builtin/align.ts'; + import { walk } from '../model/query.ts'; + import { isInteractive } from '../selection/rules.ts'; ++import { roundToPixelsCommand } from './builtin/pixels.ts'; + + /** A command as the palette lists it, by its title. */ + export interface PaletteEntry { +@@ -142,6 +143,14 @@ export const BUILTIN_ENTRIES: readonly PaletteEntry[] = [ + align('Align bottom', 'bottom'), + distribute('Distribute horizontally', 'horizontal'), + distribute('Distribute vertically', 'vertical'), ++ { ++ title: 'Round to pixels', ++ enabled: hasSelection, ++ run(ctx) { ++ const c = roundToPixelsCommand(ctx.doc, ctx.selection.ids()); ++ if (c) ctx.dispatch(c, { label: 'Round to pixels' }); ++ }, ++ }, + { + title: 'Hide', + enabled: hasSelection, diff --git a/benchmarks/proof-build/hidden/_global/global-rules.test.ts b/benchmarks/proof-build/hidden/_global/global-rules.test.ts index 0c948bfd..e8a2cf1b 100644 --- a/benchmarks/proof-build/hidden/_global/global-rules.test.ts +++ b/benchmarks/proof-build/hidden/_global/global-rules.test.ts @@ -1,7 +1,7 @@ // Invariants that hold rules only Proof states: a shipped migration step // never changes its output (R3), and one palette command run is at most -// one history entry (R1). grade.mjs runs this file only with proof-only -// and stale tasks (see globals.json). The expected values were computed +// one history entry (R1). grade.mjs runs this file only with proof-only, +// proof-only-hard and stale tasks (see globals.json). The expected values were computed // from the editor as it was before any task. import { test } from 'node:test'; import assert from 'node:assert/strict'; diff --git a/benchmarks/proof-build/hidden/_global/globals.json b/benchmarks/proof-build/hidden/_global/globals.json index 635c3e74..543e99ba 100644 --- a/benchmarks/proof-build/hidden/_global/globals.json +++ b/benchmarks/proof-build/hidden/_global/globals.json @@ -1 +1 @@ -{ "global-rules.test.ts": ["proof-only", "stale"] } +{ "global-rules.test.ts": ["proof-only", "stale", "proof-only-hard"] } diff --git a/benchmarks/proof-build/results/haiku-suspect-compare.json b/benchmarks/proof-build/results/haiku-suspect-compare.json new file mode 100644 index 00000000..1353a255 --- /dev/null +++ b/benchmarks/proof-build/results/haiku-suspect-compare.json @@ -0,0 +1,2282 @@ +{ + "work": "/p5/runs-sus", + "runs": ["u1", "u2", "u3"], + "split": "all", + "base": "none", + "flags": { + "brief": { + "sessions": 36, + "excluded": 2, + "left_folder": 2, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 19, + "edited_test_support": 0 + }, + "dump": { + "sessions": 36, + "excluded": 2, + "left_folder": 2, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 21, + "edited_test_support": 0 + }, + "find": { + "sessions": 36, + "excluded": 1, + "left_folder": 1, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 20, + "edited_test_support": 0 + }, + "findcode": { + "sessions": 36, + "excluded": 1, + "left_folder": 1, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 20, + "edited_test_support": 0 + }, + "none": { + "sessions": 36, + "excluded": 3, + "left_folder": 3, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 20, + "edited_test_support": 0 + }, + "skill": { + "sessions": 36, + "excluded": 2, + "left_folder": 2, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 19, + "edited_test_support": 0 + } + }, + "droppedPairs": [ + "B-S2#u1", + "B-S2#u2", + "B-S4#u1", + "B-S4#u3", + "B-V1#u1", + "B-V1#u3", + "B-V2#u2", + "B-V5#u1", + "L-S1#u2" + ], + "holm_primary": { + "stale:dump": 0.1565, + "stale:skill": 0.25, + "stale:find": 0.75, + "stale:brief": 1, + "stale:findcode": 1 + }, + "holm_all_green": { + "dump": 0.005, + "find": 0.0624, + "brief": 0.1875, + "skill": 0.1875, + "findcode": 0.25 + }, + "comparisons": [ + { + "arm": "skill", + "base": "none", + "build": { + "pairs": 19, + "all_green": { + "arm": 10, + "base": 16, + "wins": ["B-V3#u3"], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-S1#u2", + "B-V3#u2", + "B-S1#u3", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 0, + "base": 3, + "p": 0.25 + } + }, + "acceptance": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 10, + "base": 16, + "wins": ["B-V3#u3"], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-S1#u2", + "B-V3#u2", + "B-S1#u3", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 0, + "base": 3, + "p": 0.25 + } + }, + "constraints_raw": { + "arm": 10, + "base": 16, + "wins": ["B-V3#u3"], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-S1#u2", + "B-V3#u2", + "B-S1#u3", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.0703, + "taskSignTest": { + "arm": 0, + "base": 3, + "p": 0.25 + } + }, + "regressions": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 3, + "score": { + "arm": 0.759, + "base": 0.845 + }, + "wins": 2, + "losses": 1, + "wilcoxon": { + "n": 3, + "w_plus": 3, + "p": 1 + }, + "avoid_violations": { + "arm": 4, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 3903, + "base": 3701, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 57406, + "base": 44408, + "ratio": 1.219, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 50851, + "base": 40714, + "ratio": 1.23, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": 887923, + "base": 586635, + "ratio": 1.426, + "armCheaper": 2 + }, + "tool_calls": { + "arm": 21, + "base": 18 + }, + "turns": { + "arm": 18, + "base": 16 + }, + "wall_ms": { + "arm": 156617, + "base": 122602, + "ratio": 1.103, + "armCheaper": 4 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 2, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3865, + "base": 3667, + "ratio": 1.054, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 60574, + "base": 46727, + "ratio": 1.178, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 56701, + "base": 43057, + "ratio": 1.188, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 414799, + "base": 266867, + "ratio": 1.34, + "armCheaper": 2 + }, + "tool_calls": { + "arm": 12, + "base": 11 + }, + "turns": { + "arm": 10, + "base": 7 + }, + "wall_ms": { + "arm": 114402, + "base": 85093, + "ratio": 1.209, + "armCheaper": 2 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 2, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 11, + "all_green": { + "arm": 7, + "base": 8, + "wins": ["B-V3#u3"], + "losses": ["B-V3#u2", "B-V4#u3"], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "acceptance": { + "arm": 11, + "base": 11, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 7, + "base": 8, + "wins": ["B-V3#u3"], + "losses": ["B-V3#u2", "B-V4#u3"], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "constraints_raw": { + "arm": 7, + "base": 8, + "wins": ["B-V3#u3"], + "losses": ["B-V3#u2", "B-V4#u3"], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 1, + "score": { + "arm": 0.786, + "base": 0.643 + }, + "wins": 2, + "losses": 0, + "wilcoxon": { + "n": 2, + "w_plus": 3, + "p": 0.5 + }, + "avoid_violations": { + "arm": 2, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3913, + "base": 3710, + "ratio": 1.053, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 60342, + "base": 48705, + "ratio": 1.193, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 56425, + "base": 45005, + "ratio": 1.204, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": 905144, + "base": 589948, + "ratio": 1.365, + "armCheaper": 1 + }, + "wall_ms": { + "arm": 161787, + "base": 135916, + "ratio": 1.057, + "armCheaper": 2 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-S1#u1", "B-S3#u1", "B-S1#u2", "B-S1#u3", "B-S3#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-S1#u1", "B-S3#u1", "B-S1#u2", "B-S1#u3", "B-S3#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-S1#u1", "B-S3#u1", "B-S1#u2", "B-S1#u3", "B-S3#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 2, + "score": { + "arm": 0.741, + "base": 0.979 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 2, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3878, + "base": 3686, + "ratio": 1.054, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 59369, + "base": 41657, + "ratio": 1.215, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 55504, + "base": 37993, + "ratio": 1.227, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 581582, + "base": 328081, + "ratio": 1.437, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 114082, + "base": 82290, + "ratio": 1.309, + "armCheaper": 4 + } + } + } + } + }, + { + "arm": "brief", + "base": "none", + "build": { + "pairs": 19, + "all_green": { + "arm": 11, + "base": 16, + "wins": [], + "losses": ["B-V3#u2", "B-V4#u2", "B-S1#u3", "B-S3#u3", "B-V4#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "acceptance": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 11, + "base": 16, + "wins": [], + "losses": ["B-V3#u2", "B-V4#u2", "B-S1#u3", "B-S3#u3", "B-V4#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "constraints_raw": { + "arm": 11, + "base": 16, + "wins": [], + "losses": ["B-V3#u2", "B-V4#u2", "B-S1#u3", "B-S3#u3", "B-V4#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "regressions": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 3, + "score": { + "arm": 0.861, + "base": 0.845 + }, + "wins": 1, + "losses": 1, + "wilcoxon": { + "n": 2, + "w_plus": 2, + "p": 1 + }, + "avoid_violations": { + "arm": 2, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 16794, + "base": 3701, + "ratio": 4.538, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 65905, + "base": 44408, + "ratio": 1.486, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 49062, + "base": 40714, + "ratio": 1.087, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 878996, + "base": 586635, + "ratio": 1.654, + "armCheaper": 1 + }, + "tool_calls": { + "arm": 19, + "base": 18 + }, + "turns": { + "arm": 16, + "base": 16 + }, + "wall_ms": { + "arm": 113372, + "base": 122602, + "ratio": 1.042, + "armCheaper": 9 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 16762, + "base": 3667, + "ratio": 4.569, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 71660, + "base": 46727, + "ratio": 1.395, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 54884, + "base": 43057, + "ratio": 1.143, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 427566, + "base": 266867, + "ratio": 1.777, + "armCheaper": 2 + }, + "tool_calls": { + "arm": 12, + "base": 11 + }, + "turns": { + "arm": 9, + "base": 7 + }, + "wall_ms": { + "arm": 107327, + "base": 85093, + "ratio": 1.196, + "armCheaper": 2 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 11, + "all_green": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 11, + "base": 11, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 1, + "score": { + "arm": 0.714, + "base": 0.643 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 2, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 16807, + "base": 3710, + "ratio": 4.53, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 73828, + "base": 48705, + "ratio": 1.379, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 57016, + "base": 45005, + "ratio": 1.08, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 886069, + "base": 589948, + "ratio": 1.709, + "armCheaper": 1 + }, + "wall_ms": { + "arm": 117488, + "base": 135916, + "ratio": 1.016, + "armCheaper": 7 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 6, + "base": 8, + "wins": [], + "losses": ["B-S1#u3", "B-S3#u3"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 6, + "base": 8, + "wins": [], + "losses": ["B-S1#u3", "B-S3#u3"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 6, + "base": 8, + "wins": [], + "losses": ["B-S1#u3", "B-S3#u3"], + "p": 0.5, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 2, + "score": { + "arm": 0.958, + "base": 0.979 + }, + "wins": 0, + "losses": 1, + "wilcoxon": { + "n": 1, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 16777, + "base": 3686, + "ratio": 4.553, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 63175, + "base": 41657, + "ratio": 1.486, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 46413, + "base": 37993, + "ratio": 1.166, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 574750, + "base": 328081, + "ratio": 1.701, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 100402, + "base": 82290, + "ratio": 1.226, + "armCheaper": 4 + } + } + } + } + }, + { + "arm": "find", + "base": "none", + "build": { + "pairs": 19, + "all_green": { + "arm": 9, + "base": 16, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-V4#u1", + "B-V3#u2", + "B-V4#u2", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.0156, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "acceptance": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 9, + "base": 16, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-V4#u1", + "B-V3#u2", + "B-V4#u2", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.0156, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "constraints_raw": { + "arm": 9, + "base": 16, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-V4#u1", + "B-V3#u2", + "B-V4#u2", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.0156, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "regressions": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 3, + "score": { + "arm": 0.914, + "base": 0.845 + }, + "wins": 2, + "losses": 1, + "wilcoxon": { + "n": 3, + "w_plus": 4, + "p": 0.75 + }, + "avoid_violations": { + "arm": 2, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 3956, + "base": 3701, + "ratio": 1.068, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 60179, + "base": 44408, + "ratio": 1.111, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 49048, + "base": 40714, + "ratio": 1.116, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 783700, + "base": 586635, + "ratio": 1.288, + "armCheaper": 5 + }, + "tool_calls": { + "arm": 22, + "base": 18 + }, + "turns": { + "arm": 18, + "base": 16 + }, + "wall_ms": { + "arm": 137620, + "base": 122602, + "ratio": 1.192, + "armCheaper": 3 + }, + "proof_reads": { + "arm": 4, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3921, + "base": 3667, + "ratio": 1.069, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 56717, + "base": 46727, + "ratio": 1.16, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 52799, + "base": 43057, + "ratio": 1.167, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 380783, + "base": 266867, + "ratio": 1.355, + "armCheaper": 2 + }, + "tool_calls": { + "arm": 13, + "base": 11 + }, + "turns": { + "arm": 10, + "base": 7 + }, + "wall_ms": { + "arm": 115270, + "base": 85093, + "ratio": 1.149, + "armCheaper": 2 + }, + "proof_reads": { + "arm": 5, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 11, + "all_green": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V4#u1", "B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 11, + "base": 11, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V4#u1", "B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 4, + "base": 8, + "wins": [], + "losses": ["B-V4#u1", "B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.125, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 1, + "score": { + "arm": 0.857, + "base": 0.643 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3972, + "base": 3710, + "ratio": 1.068, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 62720, + "base": 48705, + "ratio": 1.194, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 58726, + "base": 45005, + "ratio": 1.203, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 815675, + "base": 589948, + "ratio": 1.44, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 159074, + "base": 135916, + "ratio": 1.089, + "armCheaper": 2 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S1#u1", "B-S3#u1", "B-S3#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S1#u1", "B-S3#u1", "B-S3#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-S1#u1", "B-S3#u1", "B-S3#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 2, + "score": { + "arm": 0.952, + "base": 0.979 + }, + "wins": 1, + "losses": 1, + "wilcoxon": { + "n": 2, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3938, + "base": 3686, + "ratio": 1.069, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 51851, + "base": 41657, + "ratio": 1.093, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 47904, + "base": 37993, + "ratio": 1.096, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 459232, + "base": 328081, + "ratio": 1.288, + "armCheaper": 5 + }, + "wall_ms": { + "arm": 120184, + "base": 82290, + "ratio": 1.319, + "armCheaper": 3 + } + } + } + } + }, + { + "arm": "findcode", + "base": "none", + "build": { + "pairs": 19, + "all_green": { + "arm": 13, + "base": 16, + "wins": [], + "losses": ["B-V4#u1", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "acceptance": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 13, + "base": 16, + "wins": [], + "losses": ["B-V4#u1", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "constraints_raw": { + "arm": 13, + "base": 16, + "wins": [], + "losses": ["B-V4#u1", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "regressions": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 3, + "score": { + "arm": 0.8, + "base": 0.845 + }, + "wins": 2, + "losses": 1, + "wilcoxon": { + "n": 3, + "w_plus": 3, + "p": 1 + }, + "avoid_violations": { + "arm": 3, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 4010, + "base": 3701, + "ratio": 1.083, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 53652, + "base": 44408, + "ratio": 1.178, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 49252, + "base": 40714, + "ratio": 1.184, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 631629, + "base": 586635, + "ratio": 1.367, + "armCheaper": 4 + }, + "tool_calls": { + "arm": 19, + "base": 18 + }, + "turns": { + "arm": 16, + "base": 16 + }, + "wall_ms": { + "arm": 154212, + "base": 122602, + "ratio": 1.283, + "armCheaper": 3 + }, + "proof_reads": { + "arm": 11, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3980, + "base": 3667, + "ratio": 1.084, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 59311, + "base": 46727, + "ratio": 1.175, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 55332, + "base": 43057, + "ratio": 1.181, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 478229, + "base": 266867, + "ratio": 1.651, + "armCheaper": 2 + }, + "tool_calls": { + "arm": 14, + "base": 11 + }, + "turns": { + "arm": 12, + "base": 7 + }, + "wall_ms": { + "arm": 132700, + "base": 85093, + "ratio": 1.497, + "armCheaper": 1 + }, + "proof_reads": { + "arm": 11, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 11, + "all_green": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V4#u1", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "acceptance": { + "arm": 11, + "base": 11, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V4#u1", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + }, + "constraints_raw": { + "arm": 5, + "base": 8, + "wins": [], + "losses": ["B-V4#u1", "B-V4#u2", "B-V4#u3"], + "p": 0.25, + "taskSignTest": { + "arm": 0, + "base": 1, + "p": 1 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 1, + "score": { + "arm": 0.714, + "base": 0.643 + }, + "wins": 1, + "losses": 0, + "wilcoxon": { + "n": 1, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 2, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 4023, + "base": 3710, + "ratio": 1.084, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 57792, + "base": 48705, + "ratio": 1.162, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 53785, + "base": 45005, + "ratio": 1.168, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 749176, + "base": 589948, + "ratio": 1.395, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 158594, + "base": 135916, + "ratio": 1.248, + "armCheaper": 2 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 2, + "score": { + "arm": 0.857, + "base": 0.979 + }, + "wins": 1, + "losses": 1, + "wilcoxon": { + "n": 2, + "w_plus": 1, + "p": 1 + }, + "avoid_violations": { + "arm": 1, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3992, + "base": 3686, + "ratio": 1.083, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 49158, + "base": 41657, + "ratio": 1.203, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 43856, + "base": 37993, + "ratio": 1.212, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 551558, + "base": 328081, + "ratio": 1.648, + "armCheaper": 3 + }, + "wall_ms": { + "arm": 136744, + "base": 82290, + "ratio": 1.661, + "armCheaper": 2 + } + } + } + } + }, + { + "arm": "dump", + "base": "none", + "build": { + "pairs": 19, + "all_green": { + "arm": 5, + "base": 16, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-V3#u1", + "B-V4#u1", + "B-S1#u2", + "B-S3#u2", + "B-V3#u2", + "B-V4#u2", + "B-S1#u3", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.001, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "acceptance": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 5, + "base": 16, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-V3#u1", + "B-V4#u1", + "B-S1#u2", + "B-S3#u2", + "B-V3#u2", + "B-V4#u2", + "B-S1#u3", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.001, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "constraints_raw": { + "arm": 5, + "base": 16, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-V3#u1", + "B-V4#u1", + "B-S1#u2", + "B-S3#u2", + "B-V3#u2", + "B-V4#u2", + "B-S1#u3", + "B-S3#u3", + "B-V4#u3" + ], + "p": 0.001, + "taskSignTest": { + "arm": 0, + "base": 4, + "p": 0.125 + } + }, + "regressions": { + "arm": 19, + "base": 19, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 5, + "unjudged": 3, + "score": { + "arm": 0.721, + "base": 0.845 + }, + "wins": 3, + "losses": 2, + "wilcoxon": { + "n": 5, + "w_plus": 6, + "p": 0.8125 + }, + "avoid_violations": { + "arm": 5, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 42170, + "base": 3701, + "ratio": 11.394, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 85881, + "base": 44408, + "ratio": 1.944, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 43705, + "base": 40714, + "ratio": 1.042, + "armCheaper": 6 + }, + "processed_tokens": { + "arm": 1068485, + "base": 586635, + "ratio": 2.172, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 19, + "base": 18 + }, + "turns": { + "arm": 14, + "base": 16 + }, + "wall_ms": { + "arm": 129417, + "base": 122602, + "ratio": 1.12, + "armCheaper": 6 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 42138, + "base": 3667, + "ratio": 11.489, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 88271, + "base": 46727, + "ratio": 1.925, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 46127, + "base": 43057, + "ratio": 1.135, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 545681, + "base": 266867, + "ratio": 1.956, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 12, + "base": 11 + }, + "turns": { + "arm": 7, + "base": 7 + }, + "wall_ms": { + "arm": 117304, + "base": 85093, + "ratio": 1.212, + "armCheaper": 0 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "code-visible": { + "build": { + "pairs": 11, + "all_green": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-V3#u1", "B-V4#u1", "B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 11, + "base": 11, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-V3#u1", "B-V4#u1", "B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 3, + "base": 8, + "wins": [], + "losses": ["B-V3#u1", "B-V4#u1", "B-V3#u2", "B-V4#u2", "B-V4#u3"], + "p": 0.0625, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 2, + "unjudged": 1, + "score": { + "arm": 0.786, + "base": 0.643 + }, + "wins": 2, + "losses": 0, + "wilcoxon": { + "n": 2, + "w_plus": 3, + "p": 0.5 + }, + "avoid_violations": { + "arm": 2, + "base": 2 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 42180, + "base": 3710, + "ratio": 11.371, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 96756, + "base": 48705, + "ratio": 1.979, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 54588, + "base": 45005, + "ratio": 1.196, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 1064322, + "base": 589948, + "ratio": 2.286, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 143252, + "base": 135916, + "ratio": 1.16, + "armCheaper": 4 + } + } + }, + "stale": { + "build": { + "pairs": 8, + "all_green": { + "arm": 2, + "base": 8, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-S1#u2", + "B-S3#u2", + "B-S1#u3", + "B-S3#u3" + ], + "p": 0.0313, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "acceptance": { + "arm": 8, + "base": 8, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 2, + "base": 8, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-S1#u2", + "B-S3#u2", + "B-S1#u3", + "B-S3#u3" + ], + "p": 0.0313, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + }, + "constraints_raw": { + "arm": 2, + "base": 8, + "wins": [], + "losses": [ + "B-S1#u1", + "B-S3#u1", + "B-S1#u2", + "B-S3#u2", + "B-S1#u3", + "B-S3#u3" + ], + "p": 0.0313, + "taskSignTest": { + "arm": 0, + "base": 2, + "p": 0.5 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 2, + "score": { + "arm": 0.679, + "base": 0.979 + }, + "wins": 1, + "losses": 2, + "wilcoxon": { + "n": 3, + "w_plus": 1, + "p": 0.5 + }, + "avoid_violations": { + "arm": 3, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 42146, + "base": 3686, + "ratio": 11.434, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 83290, + "base": 41657, + "ratio": 1.944, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 41148, + "base": 37993, + "ratio": 1.032, + "armCheaper": 6 + }, + "processed_tokens": { + "arm": 760902, + "base": 328081, + "ratio": 2.066, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 117743, + "base": 82290, + "ratio": 1.152, + "armCheaper": 2 + } + } + } + } + } + ] +} diff --git a/benchmarks/proof-build/results/haiku-suspect-sessions.json b/benchmarks/proof-build/results/haiku-suspect-sessions.json new file mode 100644 index 00000000..608bac15 --- /dev/null +++ b/benchmarks/proof-build/results/haiku-suspect-sessions.json @@ -0,0 +1,4652 @@ +[ + { + "run": "u1", + "arm": "brief", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 61425, + "processed_tokens": 776634, + "wall_ms": 94417, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 25, + "context_tokens": 58594, + "processed_tokens": 1030700, + "wall_ms": 139557, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 15, + "context_tokens": 42990, + "processed_tokens": 574750, + "wall_ms": 91546, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 21, + "context_tokens": 57529, + "processed_tokens": 763501, + "wall_ms": 180508, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 22, + "context_tokens": 69485, + "processed_tokens": 1020014, + "wall_ms": 169505, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 16, + "context_tokens": 73944, + "processed_tokens": 1075416, + "wall_ms": 134289, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 91179, + "processed_tokens": 1133911, + "wall_ms": 190897, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 22, + "context_tokens": 76326, + "processed_tokens": 1370710, + "wall_ms": 242056, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 48527, + "processed_tokens": 548432, + "wall_ms": 106976, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 10, + "context_tokens": 72870, + "processed_tokens": 361485, + "wall_ms": 115994, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 9, + "context_tokens": 70450, + "processed_tokens": 442523, + "wall_ms": 100896, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "brief", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 14, + "context_tokens": 55615, + "processed_tokens": 412608, + "wall_ms": 92591, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 23, + "context_tokens": 93967, + "processed_tokens": 1363164, + "wall_ms": 162269, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 25, + "context_tokens": 83605, + "processed_tokens": 1172806, + "wall_ms": 156764, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 66059, + "processed_tokens": 907460, + "wall_ms": 112990, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 31, + "context_tokens": 106487, + "processed_tokens": 1514368, + "wall_ms": 234001, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 35, + "context_tokens": 87905, + "processed_tokens": 1319953, + "wall_ms": 165212, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 94861, + "processed_tokens": 1068485, + "wall_ms": 127606, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 24, + "tool_calls": 23, + "context_tokens": 118860, + "processed_tokens": 2557216, + "wall_ms": 206885, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 98807, + "processed_tokens": 1916550, + "wall_ms": 182502, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 76769, + "processed_tokens": 894843, + "wall_ms": 108637, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.2857, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 24, + "context_tokens": 86950, + "processed_tokens": 577193, + "wall_ms": 145717, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 72588, + "processed_tokens": 462937, + "wall_ms": 104772, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "dump", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 91151, + "processed_tokens": 514169, + "wall_ms": 114614, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 19, + "tool_calls": 19, + "context_tokens": 60179, + "processed_tokens": 822473, + "wall_ms": 120184, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "timed_out": false, + "turns": 19, + "tool_calls": 27, + "context_tokens": 48535, + "processed_tokens": 717802, + "wall_ms": 215964, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 19, + "context_tokens": 38899, + "processed_tokens": 460485, + "wall_ms": 137620, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 3, + "timed_out": false, + "turns": 23, + "tool_calls": 24, + "context_tokens": 54292, + "processed_tokens": 1019551, + "wall_ms": 230912, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 24, + "tool_calls": 24, + "context_tokens": 56456, + "processed_tokens": 1013797, + "wall_ms": 208413, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 20, + "tool_calls": 25, + "context_tokens": 71243, + "processed_tokens": 1148884, + "wall_ms": 221041, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 16, + "tool_calls": 21, + "context_tokens": 63567, + "processed_tokens": 728775, + "wall_ms": 168355, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 9, + "timed_out": false, + "turns": 21, + "tool_calls": 45, + "context_tokens": 75700, + "processed_tokens": 1221732, + "wall_ms": 290344, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 14, + "tool_calls": 22, + "context_tokens": 40304, + "processed_tokens": 488113, + "wall_ms": 146508, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 60892, + "processed_tokens": 436502, + "wall_ms": 123958, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 14, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 58760, + "processed_tokens": 339867, + "wall_ms": 127885, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "find", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 50432, + "processed_tokens": 377785, + "wall_ms": 102367, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 53652, + "processed_tokens": 540176, + "wall_ms": 156852, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 8, + "timed_out": false, + "turns": 23, + "tool_calls": 22, + "context_tokens": 47897, + "processed_tokens": 920499, + "wall_ms": 224411, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 9, + "timed_out": false, + "turns": 12, + "tool_calls": 14, + "context_tokens": 36659, + "processed_tokens": 392269, + "wall_ms": 100984, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 63009, + "processed_tokens": 879033, + "wall_ms": 270403, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 13, + "timed_out": false, + "turns": 27, + "tool_calls": 26, + "context_tokens": 63357, + "processed_tokens": 1407232, + "wall_ms": 286918, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 17, + "tool_calls": 22, + "context_tokens": 52503, + "processed_tokens": 747709, + "wall_ms": 170288, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 14, + "tool_calls": 19, + "context_tokens": 58479, + "processed_tokens": 631629, + "wall_ms": 154212, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 12, + "timed_out": false, + "turns": 24, + "tool_calls": 23, + "context_tokens": 82352, + "processed_tokens": 1499036, + "wall_ms": 380111, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 9, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 56820, + "processed_tokens": 709116, + "wall_ms": 180628, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.5714, + "excluded": false, + "proof_reads": 9, + "timed_out": false, + "turns": 13, + "tool_calls": 20, + "context_tokens": 80018, + "processed_tokens": 684573, + "wall_ms": 211408, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 16, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 63615, + "processed_tokens": 556284, + "wall_ms": 163802, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "findcode", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 14, + "timed_out": false, + "turns": 10, + "tool_calls": 14, + "context_tokens": 57105, + "processed_tokens": 400174, + "wall_ms": 128102, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 39965, + "processed_tokens": 281904, + "wall_ms": 59645, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 32, + "context_tokens": 47005, + "processed_tokens": 743751, + "wall_ms": 174528, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 28, + "context_tokens": 35585, + "processed_tokens": 568345, + "wall_ms": 101222, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 19, + "context_tokens": 51525, + "processed_tokens": 730697, + "wall_ms": 199623, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 52895, + "processed_tokens": 880225, + "wall_ms": 212682, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 15, + "context_tokens": 44984, + "processed_tokens": 593261, + "wall_ms": 136939, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 22, + "context_tokens": 54413, + "processed_tokens": 696585, + "wall_ms": 155603, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 19, + "context_tokens": 61557, + "processed_tokens": 846340, + "wall_ms": 200771, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 22, + "context_tokens": 43320, + "processed_tokens": 615601, + "wall_ms": 282922, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 16, + "context_tokens": 69795, + "processed_tokens": 516557, + "wall_ms": 143392, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 51617, + "processed_tokens": 260136, + "wall_ms": 82290, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "none", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.5714, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 40025, + "processed_tokens": 222732, + "wall_ms": 84309, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 19, + "context_tokens": 57406, + "processed_tokens": 816717, + "wall_ms": 156617, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 24, + "tool_calls": 29, + "context_tokens": 49759, + "processed_tokens": 1051694, + "wall_ms": 195032, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 40258, + "processed_tokens": 423546, + "wall_ms": 79395, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 34, + "tool_calls": 36, + "context_tokens": 73632, + "processed_tokens": 1960295, + "wall_ms": 262977, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 26, + "tool_calls": 25, + "context_tokens": 57689, + "processed_tokens": 1194726, + "wall_ms": 201441, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 19, + "context_tokens": 52824, + "processed_tokens": 846115, + "wall_ms": 137322, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 24, + "context_tokens": 65933, + "processed_tokens": 947515, + "wall_ms": 166554, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 23, + "context_tokens": 71771, + "processed_tokens": 887923, + "wall_ms": 162299, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 19, + "context_tokens": 52207, + "processed_tokens": 643650, + "wall_ms": 192447, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.2857, + "excluded": false, + "proof_reads": 1, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 61778, + "processed_tokens": 508269, + "wall_ms": 123517, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 59369, + "processed_tokens": 317131, + "wall_ms": 103135, + "model": "claude-haiku-5-5" + }, + { + "run": "u1", + "arm": "skill", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 52929, + "processed_tokens": 389653, + "wall_ms": 114722, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 54215, + "processed_tokens": 553808, + "wall_ms": 72176, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 24, + "context_tokens": 52765, + "processed_tokens": 677847, + "wall_ms": 122057, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 11, + "context_tokens": 47505, + "processed_tokens": 407504, + "wall_ms": 67975, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 27, + "context_tokens": 66961, + "processed_tokens": 1166755, + "wall_ms": 172274, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 18, + "context_tokens": 65905, + "processed_tokens": 826154, + "wall_ms": 113372, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 18, + "context_tokens": 67324, + "processed_tokens": 1129681, + "wall_ms": 166724, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 24, + "context_tokens": 83297, + "processed_tokens": 1179916, + "wall_ms": 183069, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 17, + "context_tokens": 73711, + "processed_tokens": 854032, + "wall_ms": 121785, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 62360, + "processed_tokens": 669965, + "wall_ms": 96234, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 17, + "context_tokens": 91604, + "processed_tokens": 780924, + "wall_ms": 149867, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 65669, + "processed_tokens": 359844, + "wall_ms": 100402, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "brief", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 80230, + "processed_tokens": 659356, + "wall_ms": 117223, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 74531, + "processed_tokens": 760902, + "wall_ms": 129417, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 25, + "tool_calls": 33, + "context_tokens": 86110, + "processed_tokens": 1989431, + "wall_ms": 190105, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 11, + "context_tokens": 62156, + "processed_tokens": 628272, + "wall_ms": 64062, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 85881, + "processed_tokens": 1531429, + "wall_ms": 167929, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 18, + "context_tokens": 84746, + "processed_tokens": 920892, + "wall_ms": 157641, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 18, + "context_tokens": 79670, + "processed_tokens": 1288146, + "wall_ms": 150052, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 32, + "context_tokens": 106234, + "processed_tokens": 1965585, + "wall_ms": 218937, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 29, + "context_tokens": 122158, + "processed_tokens": 2455367, + "wall_ms": 184801, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 20, + "context_tokens": 75049, + "processed_tokens": 957016, + "wall_ms": 111110, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.2857, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 13, + "context_tokens": 108155, + "processed_tokens": 747163, + "wall_ms": 158509, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 100763, + "processed_tokens": 842546, + "wall_ms": 137175, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "dump", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 16, + "context_tokens": 69004, + "processed_tokens": 381017, + "wall_ms": 97061, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 37896, + "processed_tokens": 345511, + "wall_ms": 70992, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 30, + "tool_calls": 29, + "context_tokens": 41433, + "processed_tokens": 1006993, + "wall_ms": 156065, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 16, + "tool_calls": 22, + "context_tokens": 34486, + "processed_tokens": 504419, + "wall_ms": 110204, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 18, + "tool_calls": 26, + "context_tokens": 70010, + "processed_tokens": 931644, + "wall_ms": 179118, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 18, + "tool_calls": 24, + "context_tokens": 62310, + "processed_tokens": 877581, + "wall_ms": 168376, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: Scale selection scales sizes and positions about the centre of the selection", + "acceptance: a percent sign is fine, and smaller works too", + "acceptance: a single layer scales about its own centre", + "acceptance: what is inside a selected group or frame scales with it", + "acceptance: it is one step to undo", + "acceptance: anything but a positive number is refused" + ], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 26, + "tool_calls": 25, + "context_tokens": 52982, + "processed_tokens": 1121331, + "wall_ms": 233438, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 24, + "tool_calls": 27, + "context_tokens": 79513, + "processed_tokens": 1405065, + "wall_ms": 214466, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 20, + "tool_calls": 27, + "context_tokens": 70016, + "processed_tokens": 1091015, + "wall_ms": 169937, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 40397, + "processed_tokens": 375086, + "wall_ms": 91313, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 8, + "timed_out": false, + "turns": 9, + "tool_calls": 11, + "context_tokens": 61795, + "processed_tokens": 388074, + "wall_ms": 121971, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 62670, + "processed_tokens": 459232, + "wall_ms": 118452, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "find", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 8, + "tool_calls": 12, + "context_tokens": 54673, + "processed_tokens": 313743, + "wall_ms": 88820, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 41404, + "processed_tokens": 551558, + "wall_ms": 130442, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 22, + "tool_calls": 28, + "context_tokens": 62187, + "processed_tokens": 1164879, + "wall_ms": 212898, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 8, + "timed_out": false, + "turns": 14, + "tool_calls": 18, + "context_tokens": 34421, + "processed_tokens": 441470, + "wall_ms": 122649, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "timed_out": false, + "turns": 35, + "tool_calls": 34, + "context_tokens": 79148, + "processed_tokens": 2118170, + "wall_ms": 271130, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 18, + "tool_calls": 27, + "context_tokens": 61520, + "processed_tokens": 898547, + "wall_ms": 206100, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 8, + "timed_out": false, + "turns": 16, + "tool_calls": 17, + "context_tokens": 51355, + "processed_tokens": 665768, + "wall_ms": 163674, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 24, + "tool_calls": 23, + "context_tokens": 76789, + "processed_tokens": 1337191, + "wall_ms": 251471, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 13, + "timed_out": false, + "turns": 25, + "tool_calls": 24, + "context_tokens": 74360, + "processed_tokens": 1478656, + "wall_ms": 193297, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 13, + "timed_out": false, + "turns": 15, + "tool_calls": 16, + "context_tokens": 41004, + "processed_tokens": 532651, + "wall_ms": 125218, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 12, + "timed_out": false, + "turns": 13, + "tool_calls": 18, + "context_tokens": 74630, + "processed_tokens": 724973, + "wall_ms": 148907, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 9, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 47832, + "processed_tokens": 245747, + "wall_ms": 93159, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "findcode", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 45656, + "processed_tokens": 323775, + "wall_ms": 110226, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 34852, + "processed_tokens": 287779, + "wall_ms": 53840, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 27, + "context_tokens": 43813, + "processed_tokens": 721099, + "wall_ms": 125619, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 31971, + "processed_tokens": 328081, + "wall_ms": 49990, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 17, + "context_tokens": 53333, + "processed_tokens": 705235, + "wall_ms": 172345, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 16, + "context_tokens": 41202, + "processed_tokens": 372026, + "wall_ms": 114616, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: anything but a positive number is refused", + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 20, + "context_tokens": 53600, + "processed_tokens": 716333, + "wall_ms": 159319, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 28, + "context_tokens": 65179, + "processed_tokens": 978157, + "wall_ms": 200936, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 21, + "context_tokens": 64286, + "processed_tokens": 888143, + "wall_ms": 155207, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 18, + "context_tokens": 35973, + "processed_tokens": 374136, + "wall_ms": 76440, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 11, + "context_tokens": 48649, + "processed_tokens": 305909, + "wall_ms": 87489, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.9375, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 15, + "context_tokens": 53024, + "processed_tokens": 543528, + "wall_ms": 133314, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "none", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.7143, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 11, + "context_tokens": 41836, + "processed_tokens": 215095, + "wall_ms": 85876, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 13, + "context_tokens": 49570, + "processed_tokens": 413545, + "wall_ms": 90490, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 27, + "context_tokens": 47231, + "processed_tokens": 836944, + "wall_ms": 141468, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 13, + "context_tokens": 37758, + "processed_tokens": 457398, + "wall_ms": 90226, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 28, + "tool_calls": 27, + "context_tokens": 67176, + "processed_tokens": 1494398, + "wall_ms": 190162, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 53618, + "processed_tokens": 997491, + "wall_ms": 161589, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 23, + "context_tokens": 60927, + "processed_tokens": 1008244, + "wall_ms": 214308, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 20, + "tool_calls": 25, + "context_tokens": 80518, + "processed_tokens": 1159636, + "wall_ms": 201892, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 22, + "context_tokens": 71851, + "processed_tokens": 922365, + "wall_ms": 161985, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 17, + "context_tokens": 39083, + "processed_tokens": 384280, + "wall_ms": 78044, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 16, + "context_tokens": 70163, + "processed_tokens": 687309, + "wall_ms": 158187, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.9375, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 64422, + "processed_tokens": 439945, + "wall_ms": 124283, + "model": "claude-haiku-5-5" + }, + { + "run": "u2", + "arm": "skill", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.8571, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 46553, + "processed_tokens": 294827, + "wall_ms": 86723, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 19, + "context_tokens": 60428, + "processed_tokens": 878996, + "wall_ms": 104059, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 33, + "context_tokens": 71323, + "processed_tokens": 1343424, + "wall_ms": 141136, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 19, + "context_tokens": 54722, + "processed_tokens": 864494, + "wall_ms": 95094, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 57832, + "processed_tokens": 569365, + "wall_ms": 122048, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 65591, + "processed_tokens": 1011683, + "wall_ms": 148650, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 19, + "context_tokens": 63220, + "processed_tokens": 928578, + "wall_ms": 117752, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 21, + "context_tokens": 75819, + "processed_tokens": 1079234, + "wall_ms": 146460, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 19, + "context_tokens": 72340, + "processed_tokens": 918106, + "wall_ms": 112384, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 21, + "context_tokens": 60431, + "processed_tokens": 697819, + "wall_ms": 103079, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 15, + "context_tokens": 74880, + "processed_tokens": 633532, + "wall_ms": 117078, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 63175, + "processed_tokens": 351622, + "wall_ms": 85733, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "brief", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 14, + "context_tokens": 82140, + "processed_tokens": 779981, + "wall_ms": 113757, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a number key gives a group its own opacity and leaves its layers alone", + "constraints: a number key does what typing the percentage into Opacity does", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 17, + "context_tokens": 78525, + "processed_tokens": 883119, + "wall_ms": 116074, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 22, + "context_tokens": 84162, + "processed_tokens": 1527611, + "wall_ms": 117743, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 65376, + "processed_tokens": 716293, + "wall_ms": 59374, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 21, + "context_tokens": 80224, + "processed_tokens": 1181357, + "wall_ms": 158602, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 83372, + "processed_tokens": 972050, + "wall_ms": 131459, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 16, + "context_tokens": 98651, + "processed_tokens": 1060159, + "wall_ms": 129274, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 27, + "context_tokens": 125680, + "processed_tokens": 2226088, + "wall_ms": 225360, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 21, + "context_tokens": 108331, + "processed_tokens": 1625352, + "wall_ms": 157229, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 12, + "context_tokens": 79710, + "processed_tokens": 690158, + "wall_ms": 72137, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 11, + "context_tokens": 104270, + "processed_tokens": 583317, + "wall_ms": 170069, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 83290, + "processed_tokens": 508435, + "wall_ms": 84615, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "dump", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 12, + "context_tokens": 89591, + "processed_tokens": 698809, + "wall_ms": 119993, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 13, + "tool_calls": 16, + "context_tokens": 35911, + "processed_tokens": 419477, + "wall_ms": 92021, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 16, + "tool_calls": 21, + "context_tokens": 34578, + "processed_tokens": 474531, + "wall_ms": 121221, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 28125, + "processed_tokens": 297423, + "wall_ms": 67206, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 21, + "tool_calls": 27, + "context_tokens": 53565, + "processed_tokens": 937428, + "wall_ms": 164894, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 20, + "tool_calls": 32, + "context_tokens": 55575, + "processed_tokens": 880183, + "wall_ms": 179903, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 50607, + "processed_tokens": 783700, + "wall_ms": 131531, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 18, + "tool_calls": 26, + "context_tokens": 63129, + "processed_tokens": 847650, + "wall_ms": 150889, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 23, + "tool_calls": 30, + "context_tokens": 72756, + "processed_tokens": 1285140, + "wall_ms": 167259, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 16, + "tool_calls": 22, + "context_tokens": 37331, + "processed_tokens": 508006, + "wall_ms": 105479, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 7, + "timed_out": false, + "turns": 11, + "tool_calls": 14, + "context_tokens": 51851, + "processed_tokens": 383781, + "wall_ms": 112088, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 11, + "tool_calls": 15, + "context_tokens": 62378, + "processed_tokens": 499863, + "wall_ms": 121705, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "find", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 10, + "tool_calls": 13, + "context_tokens": 36703, + "processed_tokens": 296883, + "wall_ms": 86917, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 12, + "timed_out": false, + "turns": 13, + "tool_calls": 15, + "context_tokens": 49158, + "processed_tokens": 536562, + "wall_ms": 117330, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 14, + "timed_out": false, + "turns": 17, + "tool_calls": 20, + "context_tokens": 41171, + "processed_tokens": 589744, + "wall_ms": 136744, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 8, + "timed_out": false, + "turns": 12, + "tool_calls": 13, + "context_tokens": 39400, + "processed_tokens": 416650, + "wall_ms": 75582, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 19, + "tool_calls": 29, + "context_tokens": 69845, + "processed_tokens": 1069083, + "wall_ms": 199915, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "timed_out": false, + "turns": 19, + "tool_calls": 18, + "context_tokens": 53795, + "processed_tokens": 883449, + "wall_ms": 148519, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 8, + "timed_out": false, + "turns": 16, + "tool_calls": 18, + "context_tokens": 53262, + "processed_tokens": 750643, + "wall_ms": 111897, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 11, + "timed_out": false, + "turns": 22, + "tool_calls": 36, + "context_tokens": 81819, + "processed_tokens": 1404565, + "wall_ms": 213881, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 12, + "timed_out": false, + "turns": 20, + "tool_calls": 19, + "context_tokens": 65265, + "processed_tokens": 1009533, + "wall_ms": 162975, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 11, + "timed_out": false, + "turns": 14, + "tool_calls": 22, + "context_tokens": 54499, + "processed_tokens": 590715, + "wall_ms": 127773, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 10, + "timed_out": false, + "turns": 14, + "tool_calls": 17, + "context_tokens": 64345, + "processed_tokens": 632601, + "wall_ms": 152965, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 15, + "timed_out": false, + "turns": 14, + "tool_calls": 14, + "context_tokens": 61516, + "processed_tokens": 608667, + "wall_ms": 137298, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "findcode", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 11, + "timed_out": false, + "turns": 6, + "tool_calls": 7, + "context_tokens": 41727, + "processed_tokens": 183227, + "wall_ms": 85546, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 32334, + "processed_tokens": 325652, + "wall_ms": 51166, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 20, + "tool_calls": 28, + "context_tokens": 44408, + "processed_tokens": 747420, + "wall_ms": 122602, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 12, + "context_tokens": 27185, + "processed_tokens": 346766, + "wall_ms": 58888, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 30, + "context_tokens": 62852, + "processed_tokens": 1125644, + "wall_ms": 168870, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 25, + "context_tokens": 48435, + "processed_tokens": 685155, + "wall_ms": 151697, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 21, + "tool_calls": 20, + "context_tokens": 52426, + "processed_tokens": 881021, + "wall_ms": 134892, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 13, + "tool_calls": 17, + "context_tokens": 59036, + "processed_tokens": 577762, + "wall_ms": 140554, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 17, + "tool_calls": 24, + "context_tokens": 57369, + "processed_tokens": 828776, + "wall_ms": 140340, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 24, + "context_tokens": 44167, + "processed_tokens": 586635, + "wall_ms": 107606, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 11, + "context_tokens": 53504, + "processed_tokens": 273598, + "wall_ms": 92100, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 41657, + "processed_tokens": 221317, + "wall_ms": 73477, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "none", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 22, + "context_tokens": 41601, + "processed_tokens": 284434, + "wall_ms": 80211, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-S1", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 19, + "context_tokens": 68713, + "processed_tokens": 1006936, + "wall_ms": 112295, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-S2", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 26, + "context_tokens": 50544, + "processed_tokens": 859083, + "wall_ms": 120690, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-S3", + "kind": "build", + "category": "stale", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the shortcut table plugins read stays the frozen v1 table", + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 18, + "context_tokens": 46576, + "processed_tokens": 658375, + "wall_ms": 88095, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-S4", + "kind": "build", + "category": "stale", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 24, + "tool_calls": 28, + "context_tokens": 70399, + "processed_tokens": 1332667, + "wall_ms": 197124, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-V1", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 22, + "tool_calls": 21, + "context_tokens": 50371, + "processed_tokens": 931228, + "wall_ms": 144582, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-V2", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: scaling up and back down returns the same numbers" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 16, + "tool_calls": 16, + "context_tokens": 54751, + "processed_tokens": 721026, + "wall_ms": 120423, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-V3", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 19, + "tool_calls": 21, + "context_tokens": 71966, + "processed_tokens": 1036774, + "wall_ms": 169379, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-V4", + "kind": "build", + "category": "code-visible", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: plugin API v1 keeps its members and its shortcut table" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 27, + "tool_calls": 31, + "context_tokens": 89235, + "processed_tokens": 1792856, + "wall_ms": 228258, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "B-V5", + "kind": "build", + "category": "code-visible", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 18, + "tool_calls": 24, + "context_tokens": 67425, + "processed_tokens": 999239, + "wall_ms": 186985, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "L-S1", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 17, + "context_tokens": 64106, + "processed_tokens": 689855, + "wall_ms": 120562, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "L-S2", + "kind": "plan", + "category": "stale", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 10, + "tool_calls": 11, + "context_tokens": 75850, + "processed_tokens": 581582, + "wall_ms": 114082, + "model": "claude-haiku-5-5" + }, + { + "run": "u3", + "arm": "skill", + "task": "L-V1", + "kind": "plan", + "category": "code-visible", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 16, + "context_tokens": 48158, + "processed_tokens": 372241, + "wall_ms": 93432, + "model": "claude-haiku-5-5" + } +] diff --git a/benchmarks/proof-build/results/sonnet-hard-compare.json b/benchmarks/proof-build/results/sonnet-hard-compare.json new file mode 100644 index 00000000..d6602be9 --- /dev/null +++ b/benchmarks/proof-build/results/sonnet-hard-compare.json @@ -0,0 +1,2583 @@ +{ + "work": "/p5/runs-hard", + "runs": ["h1", "h2", "h3"], + "split": "all", + "base": "none", + "flags": { + "brief": { + "sessions": 22, + "excluded": 0, + "left_folder": 0, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 18, + "edited_test_support": 0 + }, + "dump": { + "sessions": 22, + "excluded": 0, + "left_folder": 0, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 12, + "edited_test_support": 0 + }, + "dump10x": { + "sessions": 5, + "excluded": 0, + "left_folder": 0, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 2, + "edited_test_support": 0 + }, + "find": { + "sessions": 22, + "excluded": 0, + "left_folder": 0, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 16, + "edited_test_support": 0 + }, + "findcode": { + "sessions": 22, + "excluded": 1, + "left_folder": 1, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 16, + "edited_test_support": 0 + }, + "none": { + "sessions": 22, + "excluded": 1, + "left_folder": 1, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 17, + "edited_test_support": 0 + }, + "skill": { + "sessions": 22, + "excluded": 0, + "left_folder": 0, + "hidden_read": 0, + "none_found_proof": 0, + "timed_out": 0, + "api_error": 0, + "apply_failed": 0, + "edited_existing_tests": 17, + "edited_test_support": 0 + } + }, + "droppedPairs": ["B-H6#h1", "L-H1#h1"], + "holm_primary": { + "proof-only-hard:dump": 0.0005, + "proof-only-hard:skill": 0.002, + "proof-only-hard:find": 0.002, + "proof-only-hard:brief": 0.043, + "proof-only-hard:findcode": 0.043 + }, + "holm_all_green": { + "dump": 0.0005, + "skill": 0.002, + "find": 0.002, + "brief": 0.043, + "findcode": 0.043 + }, + "comparisons": [ + { + "arm": "skill", + "base": "none", + "build": { + "pairs": 17, + "all_green": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "acceptance": { + "arm": 16, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": ["B-H4#h2"], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "constraints_raw": { + "arm": 15, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0002, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "regressions": { + "arm": 17, + "base": 17, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 3900, + "base": 3704, + "ratio": 1.054, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 23061, + "base": 19854, + "ratio": 1.183, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 19160, + "base": 16148, + "ratio": 1.219, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 192637, + "base": 162613, + "ratio": 1.035, + "armCheaper": 5 + }, + "tool_calls": { + "arm": 8, + "base": 7 + }, + "turns": { + "arm": 8, + "base": 8 + }, + "wall_ms": { + "arm": 70973, + "base": 66381, + "ratio": 1.118, + "armCheaper": 5 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3844, + "base": 3634, + "ratio": 1.056, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 44491, + "base": 27377, + "ratio": 1.323, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 40622, + "base": 23743, + "ratio": 1.363, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": 266740, + "base": 112637, + "ratio": 1.819, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 10, + "base": 5 + }, + "turns": { + "arm": 8, + "base": 5 + }, + "wall_ms": { + "arm": 99365, + "base": 48058, + "ratio": 1.245, + "armCheaper": 0 + }, + "proof_reads": { + "arm": 1, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 17, + "all_green": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "acceptance": { + "arm": 16, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": ["B-H4#h2"], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "constraints_raw": { + "arm": 15, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0002, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3898, + "base": 3700, + "ratio": 1.054, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 23236, + "base": 20006, + "ratio": 1.197, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 19338, + "base": 16302, + "ratio": 1.234, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 198705, + "base": 159408, + "ratio": 1.134, + "armCheaper": 5 + }, + "wall_ms": { + "arm": 73773, + "base": 64314, + "ratio": 1.15, + "armCheaper": 5 + } + } + } + } + }, + { + "arm": "brief", + "base": "none", + "build": { + "pairs": 17, + "all_green": { + "arm": 10, + "base": 2, + "wins": [ + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 5, + "base": 1, + "p": 0.2188 + } + }, + "acceptance": { + "arm": 16, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": ["B-H4#h2"], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 10, + "base": 2, + "wins": [ + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 5, + "base": 1, + "p": 0.2188 + } + }, + "constraints_raw": { + "arm": 11, + "base": 2, + "wins": [ + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0117, + "taskSignTest": { + "arm": 5, + "base": 1, + "p": 0.2188 + } + }, + "regressions": { + "arm": 17, + "base": 17, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 16791, + "base": 3704, + "ratio": 4.534, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 33152, + "base": 19854, + "ratio": 1.65, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 16338, + "base": 16148, + "ratio": 1.027, + "armCheaper": 8 + }, + "processed_tokens": { + "arm": 275940, + "base": 162613, + "ratio": 1.58, + "armCheaper": 2 + }, + "tool_calls": { + "arm": 8, + "base": 7 + }, + "turns": { + "arm": 8, + "base": 8 + }, + "wall_ms": { + "arm": 65027, + "base": 66381, + "ratio": 1.159, + "armCheaper": 5 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 16724, + "base": 3634, + "ratio": 4.602, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 51053, + "base": 27377, + "ratio": 1.866, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 34284, + "base": 23743, + "ratio": 1.429, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 317230, + "base": 112637, + "ratio": 2.89, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 8, + "base": 5 + }, + "turns": { + "arm": 8, + "base": 5 + }, + "wall_ms": { + "arm": 90461, + "base": 48058, + "ratio": 1.882, + "armCheaper": 1 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 17, + "all_green": { + "arm": 10, + "base": 2, + "wins": [ + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 5, + "base": 1, + "p": 0.2188 + } + }, + "acceptance": { + "arm": 16, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": ["B-H4#h2"], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 10, + "base": 2, + "wins": [ + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H6#h2", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 5, + "base": 1, + "p": 0.2188 + } + }, + "constraints_raw": { + "arm": 11, + "base": 2, + "wins": [ + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0117, + "taskSignTest": { + "arm": 5, + "base": 1, + "p": 0.2188 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 16788, + "base": 3700, + "ratio": 4.536, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 35111, + "base": 20006, + "ratio": 1.66, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 18327, + "base": 16302, + "ratio": 1.032, + "armCheaper": 9 + }, + "processed_tokens": { + "arm": 278398, + "base": 159408, + "ratio": 1.607, + "armCheaper": 2 + }, + "wall_ms": { + "arm": 77195, + "base": 64314, + "ratio": 1.202, + "armCheaper": 6 + } + } + } + } + }, + { + "arm": "find", + "base": "none", + "build": { + "pairs": 17, + "all_green": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 5, + "base": 0, + "p": 0.0625 + } + }, + "acceptance": { + "arm": 17, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": [], + "p": 0.5, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 5, + "base": 0, + "p": 0.0625 + } + }, + "constraints_raw": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 5, + "base": 0, + "p": 0.0625 + } + }, + "regressions": { + "arm": 17, + "base": 17, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 3960, + "base": 3704, + "ratio": 1.069, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 25966, + "base": 19854, + "ratio": 1.252, + "armCheaper": 1 + }, + "read_tokens": { + "arm": 21989, + "base": 16148, + "ratio": 1.297, + "armCheaper": 1 + }, + "processed_tokens": { + "arm": 225286, + "base": 162613, + "ratio": 1.183, + "armCheaper": 6 + }, + "tool_calls": { + "arm": 9, + "base": 7 + }, + "turns": { + "arm": 9, + "base": 8 + }, + "wall_ms": { + "arm": 96766, + "base": 66381, + "ratio": 1.387, + "armCheaper": 5 + }, + "proof_reads": { + "arm": 3, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3888, + "base": 3634, + "ratio": 1.07, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 23071, + "base": 27377, + "ratio": 0.843, + "armCheaper": 3 + }, + "read_tokens": { + "arm": 19183, + "base": 23743, + "ratio": 0.808, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 146912, + "base": 112637, + "ratio": 1.204, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 7, + "base": 5 + }, + "turns": { + "arm": 7, + "base": 5 + }, + "wall_ms": { + "arm": 89451, + "base": 48058, + "ratio": 1.736, + "armCheaper": 0 + }, + "proof_reads": { + "arm": 4, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 17, + "all_green": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 5, + "base": 0, + "p": 0.0625 + } + }, + "acceptance": { + "arm": 17, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": [], + "p": 0.5, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 5, + "base": 0, + "p": 0.0625 + } + }, + "constraints_raw": { + "arm": 14, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0005, + "taskSignTest": { + "arm": 5, + "base": 0, + "p": 0.0625 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 3955, + "base": 3700, + "ratio": 1.069, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 25877, + "base": 20006, + "ratio": 1.141, + "armCheaper": 4 + }, + "read_tokens": { + "arm": 21907, + "base": 16302, + "ratio": 1.158, + "armCheaper": 4 + }, + "processed_tokens": { + "arm": 209187, + "base": 159408, + "ratio": 1.193, + "armCheaper": 6 + }, + "wall_ms": { + "arm": 93109, + "base": 64314, + "ratio": 1.547, + "armCheaper": 5 + } + } + } + } + }, + { + "arm": "findcode", + "base": "none", + "build": { + "pairs": 17, + "all_green": { + "arm": 10, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 3, + "base": 0, + "p": 0.25 + } + }, + "acceptance": { + "arm": 17, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": [], + "p": 0.5, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 10, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 3, + "base": 0, + "p": 0.25 + } + }, + "constraints_raw": { + "arm": 10, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 3, + "base": 0, + "p": 0.25 + } + }, + "regressions": { + "arm": 17, + "base": 17, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 4010, + "base": 3704, + "ratio": 1.083, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 25058, + "base": 19854, + "ratio": 1.278, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 21047, + "base": 16148, + "ratio": 1.322, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 213407, + "base": 162613, + "ratio": 1.555, + "armCheaper": 4 + }, + "tool_calls": { + "arm": 9, + "base": 7 + }, + "turns": { + "arm": 9, + "base": 8 + }, + "wall_ms": { + "arm": 91713, + "base": 66381, + "ratio": 1.451, + "armCheaper": 4 + }, + "proof_reads": { + "arm": 4, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 3946, + "base": 3634, + "ratio": 1.086, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 26990, + "base": 27377, + "ratio": 0.873, + "armCheaper": 2 + }, + "read_tokens": { + "arm": 23044, + "base": 23743, + "ratio": 0.84, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 150517, + "base": 112637, + "ratio": 1.233, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 7, + "base": 5 + }, + "turns": { + "arm": 7, + "base": 5 + }, + "wall_ms": { + "arm": 106757, + "base": 48058, + "ratio": 1.478, + "armCheaper": 0 + }, + "proof_reads": { + "arm": 7, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 17, + "all_green": { + "arm": 10, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 3, + "base": 0, + "p": 0.25 + } + }, + "acceptance": { + "arm": 17, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": [], + "p": 0.5, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 10, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 3, + "base": 0, + "p": 0.25 + } + }, + "constraints_raw": { + "arm": 10, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H3#h1", + "B-H4#h1", + "B-H1#h2", + "B-H4#h2", + "B-H6#h2", + "B-H1#h3", + "B-H4#h3", + "B-H6#h3" + ], + "losses": ["B-H3#h2"], + "p": 0.0215, + "taskSignTest": { + "arm": 3, + "base": 0, + "p": 0.25 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 4006, + "base": 3700, + "ratio": 1.084, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 25282, + "base": 20006, + "ratio": 1.233, + "armCheaper": 4 + }, + "read_tokens": { + "arm": 21273, + "base": 16302, + "ratio": 1.266, + "armCheaper": 4 + }, + "processed_tokens": { + "arm": 193968, + "base": 159408, + "ratio": 1.437, + "armCheaper": 4 + }, + "wall_ms": { + "arm": 92403, + "base": 64314, + "ratio": 1.465, + "armCheaper": 4 + } + } + } + } + }, + { + "arm": "dump", + "base": "none", + "build": { + "pairs": 17, + "all_green": { + "arm": 17, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H5#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0001, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "acceptance": { + "arm": 17, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": [], + "p": 0.5, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 17, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H5#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0001, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "constraints_raw": { + "arm": 17, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H5#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0001, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "regressions": { + "arm": 17, + "base": 17, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 42168, + "base": 3704, + "ratio": 11.385, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 57282, + "base": 19854, + "ratio": 2.789, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 15141, + "base": 16148, + "ratio": 0.875, + "armCheaper": 10 + }, + "processed_tokens": { + "arm": 372543, + "base": 162613, + "ratio": 2.409, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 6, + "base": 7 + }, + "turns": { + "arm": 7, + "base": 8 + }, + "wall_ms": { + "arm": 74624, + "base": 66381, + "ratio": 1.069, + "armCheaper": 7 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 0, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": 42096, + "base": 3634, + "ratio": 11.582, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 66727, + "base": 27377, + "ratio": 2.261, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 24587, + "base": 23743, + "ratio": 0.834, + "armCheaper": 2 + }, + "processed_tokens": { + "arm": 436896, + "base": 112637, + "ratio": 3.114, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 6, + "base": 5 + }, + "turns": { + "arm": 7, + "base": 5 + }, + "wall_ms": { + "arm": 99285, + "base": 48058, + "ratio": 1.876, + "armCheaper": 0 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 17, + "all_green": { + "arm": 17, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H5#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0001, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "acceptance": { + "arm": 17, + "base": 15, + "wins": ["B-H4#h1", "B-H4#h3"], + "losses": [], + "p": 0.5, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 17, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H5#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0001, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + }, + "constraints_raw": { + "arm": 17, + "base": 2, + "wins": [ + "B-H1#h1", + "B-H2#h1", + "B-H3#h1", + "B-H4#h1", + "B-H5#h1", + "B-H1#h2", + "B-H2#h2", + "B-H4#h2", + "B-H5#h2", + "B-H6#h2", + "B-H1#h3", + "B-H2#h3", + "B-H4#h3", + "B-H5#h3", + "B-H6#h3" + ], + "losses": [], + "p": 0.0001, + "taskSignTest": { + "arm": 6, + "base": 0, + "p": 0.0313 + } + } + }, + "plan": { + "pairs": 3, + "unjudged": 0, + "score": { + "arm": 1, + "base": 0.25 + }, + "wins": 3, + "losses": 0, + "wilcoxon": { + "n": 3, + "w_plus": 6, + "p": 0.25 + }, + "avoid_violations": { + "arm": 0, + "base": 6 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "first_turn_tokens": { + "arm": 42160, + "base": 3700, + "ratio": 11.393, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 58083, + "base": 20006, + "ratio": 2.775, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 15903, + "base": 16302, + "ratio": 0.862, + "armCheaper": 12 + }, + "processed_tokens": { + "arm": 386458, + "base": 159408, + "ratio": 2.644, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 81056, + "base": 64314, + "ratio": 1.134, + "armCheaper": 7 + } + } + } + } + }, + { + "arm": "dump10x", + "base": "none", + "build": { + "pairs": 4, + "all_green": { + "arm": 4, + "base": 0, + "wins": ["B-H2#h1", "B-H3#h1", "B-H4#h1", "B-H5#h1"], + "losses": [], + "p": 0.125, + "taskSignTest": { + "arm": 4, + "base": 0, + "p": 0.125 + } + }, + "acceptance": { + "arm": 4, + "base": 3, + "wins": ["B-H4#h1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 0, + "wins": ["B-H2#h1", "B-H3#h1", "B-H4#h1", "B-H5#h1"], + "losses": [], + "p": 0.125, + "taskSignTest": { + "arm": 4, + "base": 0, + "p": 0.125 + } + }, + "constraints_raw": { + "arm": 4, + "base": 0, + "wins": ["B-H2#h1", "B-H3#h1", "B-H4#h1", "B-H5#h1"], + "losses": [], + "p": 0.125, + "taskSignTest": { + "arm": 4, + "base": 0, + "p": 0.125 + } + }, + "regressions": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 0, + "unjudged": 0, + "score": { + "arm": null, + "base": null + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 224472, + "base": 3704, + "ratio": 60.627, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 238693, + "base": 20330, + "ratio": 11.765, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 14161, + "base": 16615, + "ratio": 0.858, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 1554299, + "base": 184127, + "ratio": 8.774, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 7, + "base": 9 + }, + "turns": { + "arm": 7, + "base": 9 + }, + "wall_ms": { + "arm": 55308, + "base": 68411, + "ratio": 0.741, + "armCheaper": 3 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 0 + } + }, + "plan": { + "first_turn_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "context_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "read_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "tool_calls": { + "arm": null, + "base": null + }, + "turns": { + "arm": null, + "base": null + }, + "wall_ms": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "proof_reads": { + "arm": null, + "base": null + }, + "proof_files_read": { + "arm": null, + "base": null + }, + "proof_search_exposure": { + "arm": null, + "base": null + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 4, + "all_green": { + "arm": 4, + "base": 0, + "wins": ["B-H2#h1", "B-H3#h1", "B-H4#h1", "B-H5#h1"], + "losses": [], + "p": 0.125, + "taskSignTest": { + "arm": 4, + "base": 0, + "p": 0.125 + } + }, + "acceptance": { + "arm": 4, + "base": 3, + "wins": ["B-H4#h1"], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 1, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 0, + "wins": ["B-H2#h1", "B-H3#h1", "B-H4#h1", "B-H5#h1"], + "losses": [], + "p": 0.125, + "taskSignTest": { + "arm": 4, + "base": 0, + "p": 0.125 + } + }, + "constraints_raw": { + "arm": 4, + "base": 0, + "wins": ["B-H2#h1", "B-H3#h1", "B-H4#h1", "B-H5#h1"], + "losses": [], + "p": 0.125, + "taskSignTest": { + "arm": 4, + "base": 0, + "p": 0.125 + } + } + }, + "plan": null, + "cost": { + "first_turn_tokens": { + "arm": 224472, + "base": 3704, + "ratio": 60.627, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 238693, + "base": 20330, + "ratio": 11.765, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 14161, + "base": 16615, + "ratio": 0.858, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 1554299, + "base": 184127, + "ratio": 8.774, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 55308, + "base": 68411, + "ratio": 0.741, + "armCheaper": 3 + } + } + } + } + } + ], + "dump10x_vs_dump": { + "arm": "dump10x", + "base": "dump", + "build": { + "pairs": 4, + "all_green": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "regressions": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": { + "pairs": 0, + "unjudged": 0, + "score": { + "arm": null, + "base": null + }, + "wins": 0, + "losses": 0, + "wilcoxon": { + "n": 0, + "w_plus": 0, + "p": 1 + }, + "avoid_violations": { + "arm": 0, + "base": 0 + }, + "plan_changed_code": { + "arm": 0, + "base": 0 + } + }, + "cost": { + "build": { + "first_turn_tokens": { + "arm": 224472, + "base": 42170, + "ratio": 5.323, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 238693, + "base": 58083, + "ratio": 4.088, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 14161, + "base": 15903, + "ratio": 0.813, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 1554299, + "base": 372026, + "ratio": 3.517, + "armCheaper": 0 + }, + "tool_calls": { + "arm": 7, + "base": 6 + }, + "turns": { + "arm": 7, + "base": 7 + }, + "wall_ms": { + "arm": 55308, + "base": 79359, + "ratio": 0.727, + "armCheaper": 3 + }, + "proof_reads": { + "arm": 0, + "base": 0 + }, + "proof_files_read": { + "arm": 0, + "base": 0 + }, + "proof_search_exposure": { + "arm": 1, + "base": 1 + } + }, + "plan": { + "first_turn_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "context_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "read_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "processed_tokens": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "tool_calls": { + "arm": null, + "base": null + }, + "turns": { + "arm": null, + "base": null + }, + "wall_ms": { + "arm": null, + "base": null, + "ratio": null, + "armCheaper": 0 + }, + "proof_reads": { + "arm": null, + "base": null + }, + "proof_files_read": { + "arm": null, + "base": null + }, + "proof_search_exposure": { + "arm": null, + "base": null + } + } + }, + "byCategory": { + "proof-only-hard": { + "build": { + "pairs": 4, + "all_green": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "acceptance": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + }, + "constraints_raw": { + "arm": 4, + "base": 4, + "wins": [], + "losses": [], + "p": 1, + "taskSignTest": { + "arm": 0, + "base": 0, + "p": 1 + } + } + }, + "plan": null, + "cost": { + "first_turn_tokens": { + "arm": 224472, + "base": 42170, + "ratio": 5.323, + "armCheaper": 0 + }, + "context_tokens": { + "arm": 238693, + "base": 58083, + "ratio": 4.088, + "armCheaper": 0 + }, + "read_tokens": { + "arm": 14161, + "base": 15903, + "ratio": 0.813, + "armCheaper": 3 + }, + "processed_tokens": { + "arm": 1554299, + "base": 372026, + "ratio": 3.517, + "armCheaper": 0 + }, + "wall_ms": { + "arm": 55308, + "base": 79359, + "ratio": 0.727, + "armCheaper": 3 + } + } + } + } + } +} diff --git a/benchmarks/proof-build/results/sonnet-hard-sessions.json b/benchmarks/proof-build/results/sonnet-hard-sessions.json new file mode 100644 index 00000000..fc97d74f --- /dev/null +++ b/benchmarks/proof-build/results/sonnet-hard-sessions.json @@ -0,0 +1,3228 @@ +[ + { + "run": "pilot", + "arm": "brief", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 40907, + "processed_tokens": 337417, + "wall_ms": 103407, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "brief", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 36938, + "processed_tokens": 328398, + "wall_ms": 112283, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 4, + "context_tokens": 54584, + "processed_tokens": 271121, + "wall_ms": 57698, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "dump", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 63784, + "processed_tokens": 429586, + "wall_ms": 64271, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 25337, + "processed_tokens": 192621, + "wall_ms": 101189, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "find", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 11, + "tool_calls": 12, + "context_tokens": 31702, + "processed_tokens": 309502, + "wall_ms": 104988, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 32478, + "processed_tokens": 324512, + "wall_ms": 143801, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "findcode", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 27476, + "processed_tokens": 300255, + "wall_ms": 115882, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a plugin without export hears of a rename by id alone" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 24579, + "processed_tokens": 220326, + "wall_ms": 105679, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "none", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: the selection does not limit it", + "constraints: locked layers keep their colour; hidden ones change" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 20530, + "processed_tokens": 191754, + "wall_ms": 86399, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 24927, + "processed_tokens": 199349, + "wall_ms": 63726, + "model": "claude-sonnet-5-5" + }, + { + "run": "pilot", + "arm": "skill", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 24332, + "processed_tokens": 183616, + "wall_ms": 62637, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a plugin without export hears of a rename by id alone" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 28839, + "processed_tokens": 300578, + "wall_ms": 112468, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 28425, + "processed_tokens": 274261, + "wall_ms": 71308, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a frame made around layers has no fill and does not clip" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 8, + "context_tokens": 31173, + "processed_tokens": 182490, + "wall_ms": 46985, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 36899, + "processed_tokens": 287379, + "wall_ms": 92529, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 36078, + "processed_tokens": 280855, + "wall_ms": 63254, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 34612, + "processed_tokens": 356399, + "wall_ms": 123353, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 54490, + "processed_tokens": 305009, + "wall_ms": 101455, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "brief", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 49338, + "processed_tokens": 317230, + "wall_ms": 92821, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 54837, + "processed_tokens": 333848, + "wall_ms": 74624, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 57509, + "processed_tokens": 404450, + "wall_ms": 87062, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 58656, + "processed_tokens": 289611, + "wall_ms": 55677, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 14, + "tool_calls": 13, + "context_tokens": 65908, + "processed_tokens": 919944, + "wall_ms": 122870, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 56407, + "processed_tokens": 339602, + "wall_ms": 71655, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 62496, + "processed_tokens": 559713, + "wall_ms": 88165, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.875, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 63724, + "processed_tokens": 299459, + "wall_ms": 75643, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 73086, + "processed_tokens": 523773, + "wall_ms": 104746, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump10x", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 237854, + "processed_tokens": 1191361, + "wall_ms": 56859, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump10x", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 7, + "context_tokens": 236995, + "processed_tokens": 1184182, + "wall_ms": 44602, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump10x", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 239531, + "processed_tokens": 1917236, + "wall_ms": 53757, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump10x", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 240965, + "processed_tokens": 2167497, + "wall_ms": 74104, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "dump10x", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 238560, + "processed_tokens": 1917557, + "wall_ms": 66699, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 25788, + "processed_tokens": 169478, + "wall_ms": 96766, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 28880, + "processed_tokens": 225286, + "wall_ms": 109141, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 24499, + "processed_tokens": 203099, + "wall_ms": 65545, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 30384, + "processed_tokens": 267374, + "wall_ms": 114553, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 18384, + "processed_tokens": 153295, + "wall_ms": 70433, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 14, + "tool_calls": 14, + "context_tokens": 35912, + "processed_tokens": 435682, + "wall_ms": 137894, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 31003, + "processed_tokens": 185053, + "wall_ms": 128426, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "find", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 21564, + "processed_tokens": 146912, + "wall_ms": 89451, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 11, + "tool_calls": 12, + "context_tokens": 31478, + "processed_tokens": 314028, + "wall_ms": 107079, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 24225, + "processed_tokens": 152003, + "wall_ms": 85683, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 23661, + "processed_tokens": 143212, + "wall_ms": 66019, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 7, + "timed_out": false, + "turns": 10, + "tool_calls": 11, + "context_tokens": 30044, + "processed_tokens": 258496, + "wall_ms": 129425, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 22319, + "processed_tokens": 180295, + "wall_ms": 91713, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 30187, + "processed_tokens": 271975, + "wall_ms": 105998, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": true, + "proof_reads": 4, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 33441, + "processed_tokens": 216889, + "wall_ms": 104595, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "findcode", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 7, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 26990, + "processed_tokens": 150517, + "wall_ms": 109819, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a plugin without export hears of a rename by id alone" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 19854, + "processed_tokens": 246650, + "wall_ms": 97074, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 21217, + "processed_tokens": 243514, + "wall_ms": 91804, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a frame made around layers has no fill and does not clip" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 24858, + "processed_tokens": 205641, + "wall_ms": 74575, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: the selection does not limit it", + "constraints: locked layers keep their colour; hidden ones change" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 19442, + "processed_tokens": 162613, + "wall_ms": 62246, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 17171, + "processed_tokens": 109702, + "wall_ms": 47293, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": true, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 21703, + "processed_tokens": 217002, + "wall_ms": 71732, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.25, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 36106, + "processed_tokens": 191343, + "wall_ms": 114996, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "none", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.1667, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 26445, + "processed_tokens": 109761, + "wall_ms": 41626, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 27748, + "processed_tokens": 251103, + "wall_ms": 108520, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 23319, + "processed_tokens": 177542, + "wall_ms": 98112, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 26433, + "processed_tokens": 195566, + "wall_ms": 58705, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 21714, + "processed_tokens": 168239, + "wall_ms": 88759, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 9, + "context_tokens": 23152, + "processed_tokens": 177554, + "wall_ms": 59453, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 25044, + "processed_tokens": 153127, + "wall_ms": 67622, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.75, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 40358, + "processed_tokens": 214122, + "wall_ms": 92553, + "model": "claude-sonnet-5-5" + }, + { + "run": "h1", + "arm": "skill", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 12, + "context_tokens": 48736, + "processed_tokens": 358693, + "wall_ms": 104805, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 40517, + "processed_tokens": 342482, + "wall_ms": 103542, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 34750, + "processed_tokens": 310099, + "wall_ms": 93232, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a frame made around layers has no fill and does not clip" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 30999, + "processed_tokens": 252996, + "wall_ms": 117432, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: every fill and stroke in the first colour gets the second, whatever is selected", + "acceptance: the selection does not limit it", + "acceptance: one Undo puts the old colour back everywhere" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 32562, + "processed_tokens": 227897, + "wall_ms": 56546, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 27875, + "processed_tokens": 234875, + "wall_ms": 51144, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 35949, + "processed_tokens": 315801, + "wall_ms": 87832, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 51053, + "processed_tokens": 252157, + "wall_ms": 83081, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "brief", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 51963, + "processed_tokens": 337548, + "wall_ms": 90461, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 54665, + "processed_tokens": 505104, + "wall_ms": 118762, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 56502, + "processed_tokens": 340969, + "wall_ms": 75049, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 50427, + "processed_tokens": 367632, + "wall_ms": 59142, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 61346, + "processed_tokens": 489914, + "wall_ms": 97393, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 66157, + "processed_tokens": 710041, + "wall_ms": 109450, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 4, + "context_tokens": 57282, + "processed_tokens": 284944, + "wall_ms": 47827, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 66727, + "processed_tokens": 436896, + "wall_ms": 99285, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "dump", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 61903, + "processed_tokens": 350786, + "wall_ms": 90160, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 12, + "tool_calls": 11, + "context_tokens": 23563, + "processed_tokens": 260749, + "wall_ms": 134958, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 25966, + "processed_tokens": 215275, + "wall_ms": 146500, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 1, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 20530, + "processed_tokens": 112577, + "wall_ms": 63812, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 13, + "tool_calls": 13, + "context_tokens": 29893, + "processed_tokens": 340323, + "wall_ms": 124594, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 17309, + "processed_tokens": 104805, + "wall_ms": 61302, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 31393, + "processed_tokens": 273297, + "wall_ms": 105488, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 7, + "tool_calls": 8, + "context_tokens": 37016, + "processed_tokens": 185675, + "wall_ms": 142292, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "find", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 23071, + "processed_tokens": 135652, + "wall_ms": 83443, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 22856, + "processed_tokens": 214932, + "wall_ms": 89885, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 16, + "tool_calls": 16, + "context_tokens": 25058, + "processed_tokens": 387506, + "wall_ms": 228484, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a frame made around layers has no fill and does not clip" + ], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 25506, + "processed_tokens": 238544, + "wall_ms": 119079, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 14, + "tool_calls": 14, + "context_tokens": 31202, + "processed_tokens": 417789, + "wall_ms": 153069, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 20122, + "processed_tokens": 137841, + "wall_ms": 54336, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 29223, + "processed_tokens": 187753, + "wall_ms": 62936, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 32690, + "processed_tokens": 174178, + "wall_ms": 106757, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "findcode", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 7, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 23893, + "processed_tokens": 138882, + "wall_ms": 71022, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a plugin without export hears of a rename by id alone" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 18816, + "processed_tokens": 209700, + "wall_ms": 97336, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 24449, + "processed_tokens": 189835, + "wall_ms": 74831, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 19961, + "processed_tokens": 135273, + "wall_ms": 66381, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: locked layers keep their colour; hidden ones change" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 7, + "context_tokens": 18896, + "processed_tokens": 156203, + "wall_ms": 77665, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 16899, + "processed_tokens": 88621, + "wall_ms": 34018, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": ["constraints: shapes that touched still touch after rounding"], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 4, + "context_tokens": 21508, + "processed_tokens": 96441, + "wall_ms": 44744, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.25, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 38651, + "processed_tokens": 169933, + "wall_ms": 86618, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "none", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 0.3333, + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 27377, + "processed_tokens": 112637, + "wall_ms": 48058, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 22782, + "processed_tokens": 215078, + "wall_ms": 86642, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 21049, + "processed_tokens": 168800, + "wall_ms": 101034, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 23061, + "processed_tokens": 125928, + "wall_ms": 76572, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: every fill and stroke in the first colour gets the second, whatever is selected", + "acceptance: the selection does not limit it", + "acceptance: input it cannot read changes nothing", + "acceptance: one Undo puts the old colour back everywhere" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 22119, + "processed_tokens": 192637, + "wall_ms": 77060, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 10, + "context_tokens": 22287, + "processed_tokens": 201844, + "wall_ms": 66844, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 27073, + "processed_tokens": 278947, + "wall_ms": 80398, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 2, + "timed_out": false, + "turns": 8, + "tool_calls": 10, + "context_tokens": 44491, + "processed_tokens": 266740, + "wall_ms": 99365, + "model": "claude-sonnet-5-5" + }, + { + "run": "h2", + "arm": "skill", + "task": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "all_green": null, + "acceptance": null, + "constraints": false, + "regressions": null, + "plan_score": 1, + "excluded": false, + "proof_reads": 1, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 36212, + "processed_tokens": 204943, + "wall_ms": 59816, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "brief", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a plugin without export hears of a rename by id alone" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 29810, + "processed_tokens": 302781, + "wall_ms": 65027, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "brief", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 6, + "context_tokens": 33152, + "processed_tokens": 199420, + "wall_ms": 62617, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "brief", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 31949, + "processed_tokens": 159799, + "wall_ms": 50460, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "brief", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 36633, + "processed_tokens": 248618, + "wall_ms": 53743, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "brief", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 35472, + "processed_tokens": 275940, + "wall_ms": 45600, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "brief", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 8, + "tool_calls": 8, + "context_tokens": 40933, + "processed_tokens": 313732, + "wall_ms": 86700, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "dump", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 53913, + "processed_tokens": 326336, + "wall_ms": 42748, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "dump", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 56868, + "processed_tokens": 400373, + "wall_ms": 59956, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "dump", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 9, + "context_tokens": 55920, + "processed_tokens": 279076, + "wall_ms": 55465, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "dump", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 6, + "tool_calls": 5, + "context_tokens": 64162, + "processed_tokens": 372543, + "wall_ms": 48266, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "dump", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 9, + "context_tokens": 64871, + "processed_tokens": 630910, + "wall_ms": 88182, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "dump", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 64275, + "processed_tokens": 634951, + "wall_ms": 90646, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "find", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 22523, + "processed_tokens": 183004, + "wall_ms": 56535, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "find", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 29382, + "processed_tokens": 233361, + "wall_ms": 78492, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "find", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 31016, + "processed_tokens": 304680, + "wall_ms": 101884, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "find", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 26494, + "processed_tokens": 281052, + "wall_ms": 85003, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "find", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 7, + "tool_calls": 9, + "context_tokens": 18864, + "processed_tokens": 130440, + "wall_ms": 52355, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "find", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 5, + "timed_out": false, + "turns": 12, + "tool_calls": 12, + "context_tokens": 32975, + "processed_tokens": 351431, + "wall_ms": 127422, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "findcode", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 21710, + "processed_tokens": 137985, + "wall_ms": 61930, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "findcode", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 4, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 22866, + "processed_tokens": 200183, + "wall_ms": 69394, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "findcode", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 28138, + "processed_tokens": 288393, + "wall_ms": 93092, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "findcode", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 28821, + "processed_tokens": 268228, + "wall_ms": 102581, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "findcode", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 3, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 17954, + "processed_tokens": 130784, + "wall_ms": 60850, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "findcode", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 6, + "timed_out": false, + "turns": 9, + "tool_calls": 9, + "context_tokens": 26307, + "processed_tokens": 213407, + "wall_ms": 103078, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "none", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a plugin without export hears of a rename by id alone" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 11, + "context_tokens": 16227, + "processed_tokens": 195168, + "wall_ms": 50571, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "none", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: a step up goes to the next size on the type scale", + "constraints: a step down goes to the next size down on the type scale" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 16681, + "processed_tokens": 126201, + "wall_ms": 42801, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "none", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 20050, + "processed_tokens": 133083, + "wall_ms": 49413, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "none", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": false, + "constraints": false, + "regressions": true, + "failing": [ + "acceptance: the selection does not limit it", + "constraints: locked layers keep their colour; hidden ones change" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 23025, + "processed_tokens": 239955, + "wall_ms": 96329, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "none", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 5, + "tool_calls": 5, + "context_tokens": 16667, + "processed_tokens": 87025, + "wall_ms": 41928, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "none", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": ["constraints: shapes that touched still touch after rounding"], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 15, + "tool_calls": 14, + "context_tokens": 30654, + "processed_tokens": 418324, + "wall_ms": 140120, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "skill", + "task": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 11, + "tool_calls": 10, + "context_tokens": 24511, + "processed_tokens": 257737, + "wall_ms": 66248, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "skill", + "task": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 7, + "context_tokens": 20955, + "processed_tokens": 145535, + "wall_ms": 58942, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "skill", + "task": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 24481, + "processed_tokens": 148472, + "wall_ms": 50173, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "skill", + "task": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 10, + "tool_calls": 10, + "context_tokens": 27002, + "processed_tokens": 247648, + "wall_ms": 70973, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "skill", + "task": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "all_green": false, + "acceptance": true, + "constraints": false, + "regressions": true, + "failing": [ + "constraints: the search stays inside the top-level frame of the selected layer", + "constraints: a layer outside every frame finds only the others outside frames" + ], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 7, + "tool_calls": 6, + "context_tokens": 19716, + "processed_tokens": 138454, + "wall_ms": 45046, + "model": "claude-sonnet-5-5" + }, + { + "run": "h3", + "arm": "skill", + "task": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "all_green": true, + "acceptance": true, + "constraints": true, + "regressions": true, + "failing": [], + "excluded": false, + "proof_reads": 0, + "timed_out": false, + "turns": 9, + "tool_calls": 8, + "context_tokens": 22736, + "processed_tokens": 203798, + "wall_ms": 68282, + "model": "claude-sonnet-5-5" + } +] diff --git a/benchmarks/proof-build/scripts/compare.mjs b/benchmarks/proof-build/scripts/compare.mjs index fc119269..0ca73415 100644 --- a/benchmarks/proof-build/scripts/compare.mjs +++ b/benchmarks/proof-build/scripts/compare.mjs @@ -29,9 +29,11 @@ // empty diff would otherwise pass them; `constraints_raw` is the plain // count, for reference. // -// Primary endpoints (DESIGN.md §7): all-green on proof-only tasks (H1) -// and on stale tasks (H3), each arm against `none`, ten sign tests -// Holm-corrected together (`holm_primary`, keyed `:`). The +// Primary endpoints (DESIGN.md §7): all-green on proof-only tasks (H1), +// on stale tasks (H3) and, from v2 of the bed, on proof-only-hard tasks +// (H5), each arm against `none`, fifteen sign tests Holm-corrected +// together (`holm_primary`, keyed `:`); a run without +// proof-only-hard tasks keeps the original ten. The // pooled all-green over every build task is secondary, Holm-corrected // across the five primary arms (`holm_all_green`). `dump10x` is compared // with `dump` too, without correction. @@ -63,7 +65,7 @@ import { ARMS } from './arms.mjs'; import { listSessions, parseArgs, readJson, splitOf } from './common.mjs'; const PRIMARY = ['skill', 'brief', 'find', 'findcode', 'dump']; -const PRIMARY_CATEGORIES = ['proof-only', 'stale']; +const PRIMARY_CATEGORIES = ['proof-only', 'stale', 'proof-only-hard']; const BUILD_GROUPS = [ 'all_green', 'acceptance', diff --git a/benchmarks/proof-build/scripts/gate.mjs b/benchmarks/proof-build/scripts/gate.mjs index ce31af10..4f44eb44 100644 --- a/benchmarks/proof-build/scripts/gate.mjs +++ b/benchmarks/proof-build/scripts/gate.mjs @@ -12,13 +12,21 @@ // repository), and /_global/. The result goes to --out, which // defaults to /../tasks and is rebuilt from scratch. // 2. Static checks: -// - the categories hold the pre-registered counts (build 6/3/4/3, plan -// 3/1/2/2), and each task has the files its kind needs; +// - the categories hold the pre-registered counts (build 6/3/4/3/6, +// plan 3/1/2/2/2 for proof-only, code-visible, stale, control and +// proof-only-hard), and each task has the files its kind needs; // - a prompt names no file, path, symbol, Effort or record id; // - no feature word of a task appears in Proof: any record file in the // proof bed (and the 10x bed) or any commit message; -// - no leak word of a proof-only task appears anywhere in the none bed's -// history (every commit's code, tests, docs and message); +// - no leak word of a proof-only or proof-only-hard task appears +// anywhere in the none bed's history (every commit's code, tests, docs +// and message); +// - proof-only-hard only (its records name the feature's area on +// purpose, since they were written for planned work): the records in +// task.json's `records` are live in the proof bed, every leak word is +// in them (the rule is in Proof), and the prompt neither holds a leak +// word nor shares a run of four words with them (it does not echo +// Proof); // - the phrase that states each code-visible rule is in the none bed's tip; // - a plan rubric has 4 to 6 must-include and 2 to 4 must-avoid points, // and the task has plans/good.md, trap.md and plausible.md for the @@ -38,9 +46,25 @@ import { BENCH, git, loadTasks, parseArgs } from './common.mjs'; import { gradeDiff } from './grade.mjs'; const COUNTS = { - build: { 'proof-only': 6, 'code-visible': 5, stale: 4, control: 3 }, - plan: { 'proof-only': 3, 'code-visible': 1, stale: 2, control: 2 }, + build: { + 'proof-only': 6, + 'code-visible': 5, + stale: 4, + control: 3, + 'proof-only-hard': 6, + }, + plan: { + 'proof-only': 3, + 'code-visible': 1, + stale: 2, + control: 2, + 'proof-only-hard': 2, + }, }; +/** Categories whose rules only Proof states, checked for leaks into the none bed. */ +const PROOF_ONLY = ['proof-only', 'proof-only-hard']; +/** A prompt shares no run of this many words with its rule's records. */ +const ECHO_WORDS = 4; const GROUPS = ['acceptance', 'constraints', 'regressions']; /** What a prompt may not hold: paths, file names, symbols, Proof words. */ @@ -129,6 +153,41 @@ function proofText(bed) { ).toLowerCase()}`; } +/** + * The records of the proof bed by qualified key (`persist.con-…`), with + * their text and whether they are live, read through the bed's key map. + */ +function proofRecords(bed) { + const keymap = JSON.parse(readFileSync(`${bed}.keymap.json`, 'utf8')); + const graph = join(bed, '.flatbread-proof'); + const files = new Map(); + for (const sub of readdirSync(graph, { withFileTypes: true })) + if (sub.isDirectory()) + for (const name of readdirSync(join(graph, sub.name))) + files.set(name.replace(/\.md$/, ''), join(graph, sub.name, name)); + const out = new Map(); + for (const [key, id] of Object.entries(keymap.records ?? {})) { + const file = files.get(id); + if (!file) continue; + const text = readFileSync(file, 'utf8'); + const live = + !/^(superseded_by:|invalidated_by:|retracted: true|state: rejected)/m.test( + text + ); + out.set(key, { id, text, live }); + } + return out; +} + +/** The runs of `n` words in `text`, lower case, punctuation dropped. */ +function shingles(text, n) { + const words = text.toLowerCase().match(/[a-z0-9#']+/g) ?? []; + const out = new Set(); + for (let i = 0; i + n <= words.length; i++) + out.add(words.slice(i, i + n).join(' ')); + return out; +} + /** Everything in the none bed's history: every patch and every message. */ function noneHistory(bed) { return git(bed, 'log', '-p', '--format=%B', 'main').toLowerCase(); @@ -148,6 +207,7 @@ function staticChecks(o, tasks) { const proof = proofText(o.bed); const proof10 = o.x10 ? proofText(o.x10) : ''; const history = noneHistory(o.none); + const records = proofRecords(o.bed); for (const t of tasks) { const bad = PROMPT_BANS.filter(([re]) => re.test(t.prompt)).map( ([re, what]) => `${what} (${t.prompt.match(re)[0]})` @@ -163,7 +223,7 @@ function staticChecks(o, tasks) { return proof.includes(word) || proof10.includes(word); }); check(t.id, 'no feature word in Proof', !hits.length, hits.join(', ')); - if (t.category === 'proof-only') { + if (PROOF_ONLY.includes(t.category)) { const leaks = (t.leakWords ?? []).filter((w) => history.includes(w.toLowerCase()) ); @@ -176,6 +236,44 @@ function staticChecks(o, tasks) { leaks.join(', ') ); } + if (t.category === 'proof-only-hard') { + const keys = t.records ?? []; + const found = keys.map((k) => records.get(k)); + const missing = keys.filter((k, i) => !found[i]?.live); + check( + t.id, + `its records are live in Proof (${keys.length})`, + keys.length > 0 && !missing.length, + missing.join(', ') + ); + const said = found + .filter(Boolean) + .map((r) => r.text.toLowerCase()) + .join('\n'); + const unsaid = (t.leakWords ?? []).filter( + (w) => !said.includes(w.toLowerCase()) + ); + check( + t.id, + 'every leak word is in its records (the rule is in Proof)', + !unsaid.length, + unsaid.join(', ') + ); + const prompt = t.prompt.toLowerCase(); + const inPrompt = (t.leakWords ?? []).filter((w) => + prompt.includes(w.toLowerCase()) + ); + const recordRuns = shingles(said, ECHO_WORDS); + const echoes = [...shingles(t.prompt, ECHO_WORDS)].filter((run) => + recordRuns.has(run) + ); + check( + t.id, + `prompt does not echo its records (no leak word, no ${ECHO_WORDS}-word run)`, + !inPrompt.length && !echoes.length, + [...inPrompt, ...echoes.map((e) => `"${e}"`)].join(', ') + ); + } if (t.category === 'code-visible') { const missing = (t.visibleIn ?? []).filter(({ path, text }) => { const file = join(o.none, path); diff --git a/benchmarks/proof-build/scripts/grade.mjs b/benchmarks/proof-build/scripts/grade.mjs index c1d11392..44ab2261 100644 --- a/benchmarks/proof-build/scripts/grade.mjs +++ b/benchmarks/proof-build/scripts/grade.mjs @@ -199,8 +199,8 @@ const failed = (reason) => ({ pass: false, reason }); * The _global files that apply to `task`. `_global/globals.json` maps a * file name to the categories it runs for; a file it does not list runs * for every task. Rules only Proof states (R1, R3) are graded on - * proof-only and stale tasks only, so a control or code-visible task is - * never scored on a rule its arm had no way to see. + * proof-only, proof-only-hard and stale tasks only, so a control or + * code-visible task is never scored on a rule its arm had no way to see. */ export function globalFiles(tasksDir, task) { const dir = join(tasksDir, '_global'); diff --git a/benchmarks/proof-build/scripts/judge.mjs b/benchmarks/proof-build/scripts/judge.mjs index ccb2173d..205f1ed0 100644 --- a/benchmarks/proof-build/scripts/judge.mjs +++ b/benchmarks/proof-build/scripts/judge.mjs @@ -20,7 +20,7 @@ // plausible.md (a realistic plan written from the code alone, with no // Proof). It passes when every good plan scores at least 0.8 and every trap // plan at most 0.5; the plausible scores are reported, and on proof-only -// tasks a plausible plan scoring as high as the good one means the rubric +// and proof-only-hard tasks a plausible plan scoring as high as the good one means the rubric // does not reward what only Proof says. It exits 1 when the gate fails. // // --dry-run (also --prepare-only) writes each session's judge-input.md, @@ -398,7 +398,9 @@ async function rubricGate(o, tasks) { ); } if (o.dryRun) return; - for (const task of tasks.filter((t) => t.category === 'proof-only')) { + for (const task of tasks.filter((t) => + ['proof-only', 'proof-only-hard'].includes(t.category) + )) { const score = (w) => rows.find((r) => r.task === task.id && r.which === w)?.score; if (score('plausible') >= score('good')) diff --git a/benchmarks/proof-build/scripts/run.mjs b/benchmarks/proof-build/scripts/run.mjs index f7028603..1cf11aa7 100644 --- a/benchmarks/proof-build/scripts/run.mjs +++ b/benchmarks/proof-build/scripts/run.mjs @@ -5,7 +5,7 @@ // node scripts/run.mjs --name --bed // [--none ] [--bed10x ] // [--tasks ] [--arms none,skill,brief,find,findcode,dump] -// [--only build|plan|proof-only|code-visible|stale|control] +// [--only build|plan|proof-only|code-visible|stale|control|proof-only-hard] // [--ids B-P1,L-K1] [--jobs 8] [--timeout 1500] [--model haiku] // [--effort ] [--prepare-only] [--keep] // @@ -449,10 +449,10 @@ async function main() { const tasks = pickTasks(loadTasks(o.tasks), o); if (!tasks.length) throw new Error('No tasks picked'); // Alternate arms task by task, so a change in API speed over the run - // falls on every arm alike. + // falls on every arm alike. A task's `skipArms` (task.json) are left out. const sessions = []; for (const task of tasks) - for (const arm of o.arms) + for (const arm of o.arms.filter((a) => !task.skipArms?.includes(a))) sessions.push({ task, arm, @@ -465,7 +465,7 @@ async function main() { if (!meta?.done) todo.push(s); } console.log( - `${sessions.length} sessions (${tasks.length} tasks x ${o.arms.length} arms), ${todo.length} to run, ${o.jobs} at a time` + `${sessions.length} sessions (${tasks.length} tasks x ${o.arms.length} arms, less skipped arms), ${todo.length} to run, ${o.jobs} at a time` ); const cache = join(o.dir, '.inject'); if (o.prepareOnly) { diff --git a/benchmarks/proof-build/spec/banned.mjs b/benchmarks/proof-build/spec/banned.mjs index 9facfc9c..e7df03dd 100644 --- a/benchmarks/proof-build/spec/banned.mjs +++ b/benchmarks/proof-build/spec/banned.mjs @@ -6,10 +6,18 @@ // panel", "undo" or "layers" are allowed. Matching is case-insensitive and // by substring, so keep entries specific. // -// BANNED_IN_CODE: the wording of the six proof-only rules (R1 to R6). The -// code, tests and docs of both variants must not state a rule or its reason; -// the commit messages are covered by BANNED_IN_PROOF's scan as well, since -// gen-bed.mjs scans them with both lists. +// The proof-only-hard tasks are the exception: their rules are product +// decisions recorded before the feature existed, so their records name the +// feature's area on purpose. What Proof may not hold for them is the +// prompt's own product names (palette titles, the event name), listed +// after the older tasks' words; gate.mjs also checks that no prompt echoes +// its rule's records. +// +// BANNED_IN_CODE: the wording of the proof-only rules (R1 to R6) and of the +// proof-only-hard rules (U1 to U8). The code, tests and docs of both variants +// must not state a rule or its reason; the commit messages are covered by +// BANNED_IN_PROOF's scan as well, since gen-bed.mjs scans them with both +// lists. export const BANNED_IN_PROOF = [ 'corner radius', 'radius', @@ -70,6 +78,17 @@ export const BANNED_IN_PROOF = [ 'fuchsia', 'maroon', 'navy', + // proof-only-hard: the prompts' product names + 'layerrename', + 'increase font size', + 'decrease font size', + 'frame selection', + 'replace colour', + 'replace color', + 'select same fill', + 'round to pixels', + 'review comments', + 'hand to developers', ]; export const BANNED_IN_CODE = [ @@ -101,4 +120,51 @@ export const BANNED_IN_CODE = [ 'never goes into', 'dirty', 'marked it changed', + // U1: names and text only for plugins with export + 'northwind', + 'security review', + 'document content', + 'granted export', + 'cleared for export', + // U2: text size steps follow the type scale + 'type scale', + 'halden', + 'token check', + 'next size on', + // U3: a frame made around layers has no fill and no clipping + 'wrap in frame', + 'empty fills list', + 'clip off', + 'white fill', + 'no clipping', + // U4: document-wide edits skip locked layers, include hidden ones + 'brightline', + 'signed off', + 'legal-approved', + 'hover states', + 'sweep the whole document', + // U5: picking similar layers stays in the top-level frame + 'pick-similar', + 'top-level frame that holds', + 'outside every frame', + 'off-screen', + 'starting layer', + // U6: a whole-pixel fit rounds edges, not sizes + 'abutting', + 'pixel fit', + 'seams', + 'rounds edges', + 'whole-pixel', + // U7: comments outside the file, out of history, outliving layers + 'not deliverables', + 'working notes', + 'keyed by the document', + 'detached', + 'outlives', + 'review notes', + // U8: exporter plugins; no layer names or hidden layers in exports + 'exporter plugin', + 'export module', + 'working names', + 'client sites', ]; diff --git a/benchmarks/proof-build/spec/proof.mjs b/benchmarks/proof-build/spec/proof.mjs index 70d1382a..4a03e763 100644 --- a/benchmarks/proof-build/spec/proof.mjs +++ b/benchmarks/proof-build/spec/proof.mjs @@ -1921,8 +1921,226 @@ export const PROOF = { }, }; +// Product decisions about work that was not built yet: requests deferred +// with their agreed behaviour, Decisions accepted for planned work, and +// Constraints from customers. They sit beside the events above at the +// commits their authors were working on when they wrote them. Nothing in the +// code, tests, docs or commit messages says any of it (the proof-only-hard +// tasks, DESIGN.md §3). Only c34, which made export the one way for a plugin +// to read the document, implements one of them and carries a trailer. +const defer = (issueId, resolvedBy) => ({ + type: 'ResolveIssue', + issueId, + resolution: 'deferred', + resolvedBy, +}); + +const PLANNED = { + c11: { + events: [ + finding( + 'model.fnd-wrap-hid-layers', + 'feedback', + INES, + 'Wrapping layers in a white frame made people think layers were gone', + "From the 12 February usability sessions (five people: two agencies, three in-house teams), on the prototype editor's Wrap in frame. The new frame took the default white fill and clipping. It goes where the frontmost wrapped layer was, so its white fill painted over every layer behind it in that area; four of five said the layers they had not wrapped were deleted. Two then moved a wrapped layer past the frame's edge, lost half of it to clipping, and reported it as a bug." + ), + decision( + 'model.dec-wrap-plain-frame', + INES, + 'A frame made around existing layers starts with no fill and no clipping', + 'When core gains a way to put existing layers into a new frame, that frame is created with an empty fills list and clip off, so the canvas looks exactly as it did before. People add a fill or turn clipping on when they want them. A frame drawn on an empty canvas keeps the white fill and clipping that createFrame gives, since nothing is under it yet.\n\nRejected: the createFrame defaults, for the finding; a fill matching the page, since the page colour belongs to the shell and any fill still hides the layers behind.', + { derives_from: ['model.fnd-wrap-hid-layers'] } + ), + accept('model.dec-wrap-plain-frame'), + ], + }, + + c34: { + trailers: ['plugin.con-content-export'], + events: [ + constraint( + 'plugin.con-content-export', + 'hard', + MATEO, + 'Layer names and text reach only plugins granted export', + "Northwind's security review of plugin API v1 (they run Tessera for bank clients) counts layer names and the words in text layers as document content: their designers name layers after client projects and draft copy in text layers. A plugin they have not cleared for export must not read either, by any route. So anything that hands a plugin a layer name or text, an event payload or a return value, carries it only to plugins whose manifest has export; every other plugin gets the ids alone, in the same event. The same holds for anything else people write in a document later. documentload's document name was reviewed and stays for every plugin, since the shell already shows it in the window title. Agreed with Northwind on 19 March; their next review is in September.", + { derives_from: ['plugin.dec-capabilities'] } + ), + ], + }, + + c39: { + events: [ + finding( + 'layers.fnd-offscreen-picks', + 'feedback', + MATEO, + 'People did not see what a page-wide pick added on other screens', + "Six sessions on 25 March with people who keep a whole app in one file, one top-level frame per screen (40 to 120 screens). The prototype's pick-similar commands, which start from one selected layer and add the layers that resemble it, searched the whole page. The layers they added on other screens were off-screen at the time; four of six went on to edit the selection and changed screens they never looked at." + ), + decision( + 'layers.dec-likeness-scope', + MATEO, + 'Picking similar layers stays inside the top-level frame of the starting layer', + "Commands that grow a selection from one layer to the layers that resemble it (the same fill colour, font, size or anything else; none in core yet) look only inside the top-level frame that holds the starting layer, at any depth. A starting layer that sits on the page outside every frame looks only at the other layers outside frames.\n\nRejected: the whole page, for the finding; the starting layer's own parent, which is too narrow, since one screen's buttons sit in different groups.", + { derives_from: ['layers.fnd-offscreen-picks'] } + ), + accept('layers.dec-likeness-scope'), + ], + }, + + c45: { + events: [ + issue( + 'panel.iss-size-steps', + 'request', + PRIYA, + 'Halden wants text sizes to step up and down without typing', + "Halden's design-system team asked on the February call for a quick way to make text one step bigger or smaller. In the prototype a step was one pixel, and their token check flagged 340 text layers at sizes such as 15 and 17 that came from people stepping past the sizes they use." + ), + decision( + 'panel.dec-size-steps', + PRIYA, + 'A text size step moves to the next size on the type scale', + 'When we add stepping, one step up or down goes to the next size on this scale: 8, 9, 10, 11, 12, 14, 16, 18, 20, 24, 28, 32, 36, 48, 64, 72, 96. A size between two scale sizes goes to the next one in the direction of the step, so 13 goes up to 14 and down to 12. Below 8 a step is 1 pixel and above 96 it is 8 pixels, within the 1 to 1,000 bounds. The text box height follows, as for a typed size.\n\nRejected: one pixel per step (the prototype, and the issue); a percentage per step, which leaves the scale at once.', + { derives_from: ['panel.iss-size-steps'] } + ), + accept('panel.dec-size-steps'), + defer('panel.iss-size-steps', ['panel.dec-size-steps']), + ], + }, + + c46: { + events: [ + issue( + 'commands.iss-recolour-request', + 'request', + TOMAS, + 'Teams want to replace one colour across a document at once', + "Brightline and Halden both asked for it in April: a brand colour changes and every layer that uses it has to follow. In Brightline's trial of the prototype's recolour command it rewrote their locked, legal-approved logo, and a second pass that skipped hidden layers left the hover states of 30 buttons in the old colour." + ), + decision( + 'commands.dec-sweeping-edits', + TOMAS, + 'Edits that sweep the whole document leave locked layers alone and include hidden ones', + 'Agreed with Brightline and Halden before any of it is built: a command that changes layers all over the document rather than in the selection (changing one colour everywhere is the first) skips locked layers and everything inside a locked group or frame. Their teams lock what has been signed off: logos, brand marks, approved screens. Hidden layers are changed like any other, since they are states that will be shown again.\n\nRejected: changing everything, which rewrote the locked logo; skipping hidden and locked alike, as the canvas does for clicks, which left the hover states stale.', + { derives_from: ['commands.iss-recolour-request'] } + ), + accept('commands.dec-sweeping-edits'), + defer('commands.iss-recolour-request', ['commands.dec-sweeping-edits']), + ], + }, + + c47: { + events: [ + constraint( + 'persist.con-discussion-outside', + 'hard', + INES, + 'What people write about a design never travels inside the document file', + "Agencies deliver the document file itself to their clients. In January a prototype file reached a Halden client with its layer notes still inside, one of them reading 'do not show legal yet', and Halden's contract now says working notes are not deliverables. So notes, comments, review threads and approvals are kept apart from the document, in a store of their own keyed by the document, and files.save() never writes them. The file changes only when the design does, as with view state.", + { derives_from: ['persist.dec-viewstate-sidecar'] } + ), + ], + }, + + c48: { + events: [ + finding( + 'layers.fnd-notes-lost', + 'feedback', + MATEO, + 'Deleting a layer took its review notes with it', + 'In the prototype, notes hung off layers and went when the layer went. Designers often delete a layer and draw it again; in the March review sessions, reviewers lost the threads on 11 of 40 layers that way, answers included.' + ), + decision( + 'layers.dec-notes-outlive', + MATEO, + 'A note pinned to a layer outlives the layer', + 'When review notes come to core, a note pinned to a layer stores the layer id and the point where it sits. Deleting the layer never deletes the note: it stays at its last point on the canvas, marked detached, until someone resolves or deletes it. Undoing the delete pins it back to the layer.', + { derives_from: ['layers.fnd-notes-lost'] } + ), + accept('layers.dec-notes-outlive'), + ], + }, + + c52: { + events: [ + decision( + 'plugin.dec-exporters', + MATEO, + 'Formats other than our own are written by exporter plugins', + 'Core reads and writes Tessera files and nothing else. SVG, PNG, PDF and any other format are produced by exporter plugins that take api.export.document() and convert it, ours included, so a format ships on its own schedule and a broken exporter cannot break saving.\n\nRejected: an export module in core, as in the prototype, where three formats lived in core and most model changes broke one of them.', + { derives_from: ['plugin.dec-export-capability'] } + ), + accept('plugin.dec-exporters'), + constraint( + 'plugin.con-exports-no-names', + 'hard', + MATEO, + 'Exported artwork carries no layer names and no hidden layers', + 'Exports leave Tessera for developers, client sites and app bundles, where nobody reviews them. Layer names are working names (for Northwind, client project names), so an exporter writes no layer name into its output: no element id, class, title or comment made from one. An element that needs an id gets a generated one (l1, l2 and so on). Hidden layers are left out, with everything inside them. Halden found both in a prototype SVG export that a client put on their site.', + { + derives_from: ['plugin.con-content-export', 'plugin.dec-exporters'], + } + ), + ], + }, + + c61: { + events: [ + decision( + 'commands.dec-history-design-only', + TOMAS, + 'Undo covers the design, not what people say about it', + "History holds edits to the document only. Writing, answering, resolving or deleting a comment, note or review thread is never a history entry, and Undo never removes one: in the April sessions, an Undo after a reply in the prototype's notes panel took back a colleague's note that had arrived in between, and the colleague never knew. Those actions are undone by their own delete, by the person who wrote them.", + { derives_from: ['persist.con-discussion-outside'] } + ), + accept('commands.dec-history-design-only'), + ], + }, + + c75: { + events: [ + issue( + 'units.iss-whole-pixels', + 'request', + PRIYA, + 'Put layers on whole pixels when asked', + "Brightline's icon team asks for a command that moves and sizes layers onto whole pixels before they hand icons off, since fractional edges blur when the icons are drawn at 1x." + ), + finding( + 'units.fnd-pixel-fit-seams', + 'analysis', + PRIYA, + "The prototype's pixel fit opened seams between shapes that touched", + "The prototype rounded x, y, width and height each on its own. In Brightline's 212 icons built from abutting shapes, 61 came out with a 1 px gap or a 1 px overlap: where two shapes met at 10.5, the left one (x 0.4, width 10.1) ended at 10 and the right one started at 11.", + { derives_from: ['units.iss-whole-pixels'] } + ), + decision( + 'units.dec-round-edges', + PRIYA, + 'A whole-pixel fit rounds edges, not sizes', + "When core gets this command, it rounds each layer's left, top, right and bottom edge to the nearest whole pixel (halves up, as Math.round does) and takes width and height from the rounded edges, so shapes that touched still touch and the right edge of one stays the left edge of the next. A layer that would end up narrower or shorter than 1 px gets 1 px; a line keeps its zero height. Stored geometry is otherwise never rounded ({{id:units.con-never-round}}); this is an edit the person asks for, and it goes through resize like any other.", + { derives_from: ['units.fnd-pixel-fit-seams'] } + ), + accept('units.dec-round-edges'), + defer('units.iss-whole-pixels', ['units.dec-round-edges']), + ], + }, +}; + +for (const [key, { trailers = [], events }] of Object.entries(PLANNED)) { + const at = (PROOF[key] ??= { events: [] }); + if (trailers.length) at.trailers = [...(at.trailers ?? []), ...trailers]; + at.events.push(...events); +} + // The rules the benchmark tasks lean on (DESIGN.md §2), by the records that -// hold them. R rules live only in Proof; V rules are also visible in code. +// hold them. R rules live only in Proof; V rules are also visible in code; +// U rules are product decisions for work not built yet, which neither the +// code nor its history hints at. export const RULES = { R1: { rule: 'One user gesture is one history entry, however many nodes it touches.', @@ -1995,6 +2213,55 @@ export const RULES = { visible: true, records: ['plugin.dec-api-frozen', 'plugin.con-v1-surface'], }, + U1: { + rule: 'Layer names and text reach only plugins with the export capability; other plugins get ids alone.', + visible: false, + records: ['plugin.con-content-export'], + }, + U2: { + rule: 'A text size step goes to the next size on the type scale, not one pixel.', + visible: false, + records: ['panel.iss-size-steps', 'panel.dec-size-steps'], + }, + U3: { + rule: 'A frame made around existing layers has no fill and does not clip.', + visible: false, + records: ['model.fnd-wrap-hid-layers', 'model.dec-wrap-plain-frame'], + }, + U4: { + rule: 'Document-wide edits skip locked layers (and their contents) and include hidden ones.', + visible: false, + records: ['commands.iss-recolour-request', 'commands.dec-sweeping-edits'], + }, + U5: { + rule: "Picking similar layers stays inside the starting layer's top-level frame.", + visible: false, + records: ['layers.fnd-offscreen-picks', 'layers.dec-likeness-scope'], + }, + U6: { + rule: 'A whole-pixel fit rounds edges, not sizes, so touching shapes still touch.', + visible: false, + records: [ + 'units.iss-whole-pixels', + 'units.fnd-pixel-fit-seams', + 'units.dec-round-edges', + ], + }, + U7: { + rule: 'Comments live outside the document file, are never history entries, and outlive the layers they are pinned to.', + visible: false, + records: [ + 'persist.con-discussion-outside', + 'commands.dec-history-design-only', + 'layers.fnd-notes-lost', + 'layers.dec-notes-outlive', + ], + }, + U8: { + rule: 'Other formats are written by exporter plugins; exports carry no layer names and no hidden layers.', + visible: false, + records: ['plugin.dec-exporters', 'plugin.con-exports-no-names'], + }, }; // Proper supersessions: Proof recorded the change. diff --git a/benchmarks/proof-build/tasks/B-H1/task.json b/benchmarks/proof-build/tasks/B-H1/task.json new file mode 100644 index 00000000..a12fa744 --- /dev/null +++ b/benchmarks/proof-build/tasks/B-H1/task.json @@ -0,0 +1,22 @@ +{ + "id": "B-H1", + "kind": "build", + "category": "proof-only-hard", + "rules": ["U1"], + "records": ["plugin.con-content-export"], + "prompt": "Plugin authors have asked to hear about renames. Please add a plugin event, layerrename, that fires whenever a layer's name changes, whether through the palette, the layers panel, or an undo or redo of a rename. They have asked for { id, before, after }: the layer, its old name and its new name. List it with the other events in the plugin reference.", + "featureWords": ["layerrename"], + "leakWords": [ + "northwind", + "security review", + "document content", + "granted export", + "cleared for export" + ], + "visibleIn": [], + "reference": "Emits layerrename from the document's change events, comparing each layer's name with the one plugins last heard; the host sends { id, before, after } to plugins with the export capability and { id } to the rest.", + "naive": { + "fails": ["constraints"], + "summary": "The same event with { id, before, after } emitted to every plugin, as the existing events are, so a plugin without export reads layer names." + } +} diff --git a/benchmarks/proof-build/tasks/B-H2/task.json b/benchmarks/proof-build/tasks/B-H2/task.json new file mode 100644 index 00000000..d6c353ae --- /dev/null +++ b/benchmarks/proof-build/tasks/B-H2/task.json @@ -0,0 +1,16 @@ +{ + "id": "B-H2", + "kind": "build", + "category": "proof-only-hard", + "rules": ["U2"], + "records": ["panel.iss-size-steps", "panel.dec-size-steps"], + "prompt": "I'd like Increase font size and Decrease font size in the palette, so I can make the selected text bigger or smaller a step at a time without typing a number. They change every text layer in the selection, each from its own size, and leave other layers alone. They're offered only when the selection includes a text layer. The text box should grow or shrink to fit, as it does when I type a size, and one Undo takes a whole step back.", + "featureWords": ["increase font size", "decrease font size"], + "leakWords": ["type scale", "halden", "token check", "next size on"], + "visibleIn": [], + "reference": "Steps each selected text layer to the next size on the type scale (8, 9, 10, 11, 12, 14, 16, 18, 20, 24, 28, 32, 36, 48, 64, 72, 96; 1 px below it, 8 px above), setting the height as a typed size does, in one batch.", + "naive": { + "fails": ["constraints"], + "summary": "Adds or takes away 1 px per step, as the prototype did, so 16 goes to 17." + } +} diff --git a/benchmarks/proof-build/tasks/B-H3/task.json b/benchmarks/proof-build/tasks/B-H3/task.json new file mode 100644 index 00000000..4710c82f --- /dev/null +++ b/benchmarks/proof-build/tasks/B-H3/task.json @@ -0,0 +1,22 @@ +{ + "id": "B-H3", + "kind": "build", + "category": "proof-only-hard", + "rules": ["U3"], + "records": ["model.fnd-wrap-hid-layers", "model.dec-wrap-plain-frame"], + "prompt": "Add Frame selection to the palette, next to Group selection. It moves the selected layers into a brand-new frame drawn tightly around them, in the place of the frontmost of them, and selects that frame. The layers stay exactly where they are on the canvas and keep their order. It's offered when something is selected, and one Undo takes it all back.", + "featureWords": ["frame selection"], + "leakWords": [ + "wrap in frame", + "empty fills list", + "clip off", + "white fill", + "no clipping" + ], + "visibleIn": [], + "reference": "Builds the frame like Group selection builds a group (insert, then reorder each member in one batch), sized to the selection bounds, with an empty fills list and clip off.", + "naive": { + "fails": ["constraints"], + "summary": "The new frame takes createFrame's defaults, a white fill and clipping, which hide the layers behind it." + } +} diff --git a/benchmarks/proof-build/tasks/B-H4/task.json b/benchmarks/proof-build/tasks/B-H4/task.json new file mode 100644 index 00000000..0f835712 --- /dev/null +++ b/benchmarks/proof-build/tasks/B-H4/task.json @@ -0,0 +1,22 @@ +{ + "id": "B-H4", + "kind": "build", + "category": "proof-only-hard", + "rules": ["U4"], + "records": ["commands.iss-recolour-request", "commands.dec-sweeping-edits"], + "prompt": "I'd like Replace colour in the palette, for swapping one colour for another across a document. I type the colour to find and the colour to use, separated by a space, for example #FF0000 #00AA00. Fills and strokes in the first colour then get the second, in groups and frames too, whatever is selected. If I type something it can't read, nothing changes, and one Undo puts the old colour back everywhere.", + "featureWords": ["replace colour"], + "leakWords": [ + "brightline", + "signed off", + "legal-approved", + "hover states", + "sweep the whole document" + ], + "visibleIn": [], + "reference": "Walks the document and replaces the colour in fills and strokes in one batch, skipping locked layers and everything inside a locked group or frame; hidden layers change.", + "naive": { + "fails": ["constraints"], + "summary": "Replaces the colour on every layer, locked ones included." + } +} diff --git a/benchmarks/proof-build/tasks/B-H5/task.json b/benchmarks/proof-build/tasks/B-H5/task.json new file mode 100644 index 00000000..2816acf6 --- /dev/null +++ b/benchmarks/proof-build/tasks/B-H5/task.json @@ -0,0 +1,22 @@ +{ + "id": "B-H5", + "kind": "build", + "category": "proof-only-hard", + "rules": ["U5"], + "records": ["layers.fnd-offscreen-picks", "layers.dec-likeness-scope"], + "prompt": "Add Select same fill to the palette. With one layer selected, it selects the layers whose fill is the same colour as that one's, the original included, so I can change them together, even when they sit inside groups. It's offered only when exactly one layer with a fill is selected.", + "featureWords": ["select same fill"], + "leakWords": [ + "pick-similar", + "top-level frame that holds", + "outside every frame", + "off-screen", + "starting layer" + ], + "visibleIn": [], + "reference": "Searches only the top-level frame that holds the selected layer, at any depth, or, for a layer outside every frame, the other layers outside frames.", + "naive": { + "fails": ["constraints"], + "summary": "Searches the whole document, so matching layers on every other screen are selected too." + } +} diff --git a/benchmarks/proof-build/tasks/B-H6/task.json b/benchmarks/proof-build/tasks/B-H6/task.json new file mode 100644 index 00000000..b6c868fa --- /dev/null +++ b/benchmarks/proof-build/tasks/B-H6/task.json @@ -0,0 +1,26 @@ +{ + "id": "B-H6", + "kind": "build", + "category": "proof-only-hard", + "rules": ["U6"], + "records": [ + "units.iss-whole-pixels", + "units.fnd-pixel-fit-seams", + "units.dec-round-edges" + ], + "prompt": "Add Round to pixels to the palette, so icons come out crisp: afterwards the selected layers' X, Y, W and H are all whole numbers. It works on everything selected, including the layers inside selected groups and frames, and one Undo puts them all back.", + "featureWords": ["round to pixels"], + "leakWords": [ + "abutting", + "pixel fit", + "seams", + "rounds edges", + "whole-pixel" + ], + "visibleIn": [], + "reference": "Rounds each layer's left, top, right and bottom edges and takes width and height from the rounded edges, in one resize command, so layers that touched still touch.", + "naive": { + "fails": ["constraints"], + "summary": "Rounds x, y, width and height each on its own, which opens a 1 px gap between shapes that met at a half pixel." + } +} diff --git a/benchmarks/proof-build/tasks/L-H1/plans/good.md b/benchmarks/proof-build/tasks/L-H1/plans/good.md new file mode 100644 index 00000000..2de1976a --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H1/plans/good.md @@ -0,0 +1,8 @@ +# Plan: review comments + +1. Model. A thread is `{ id, documentKey, anchor, comments: [{ id, author, at, text }], resolved }`. The anchor is either a canvas point, or a layer id plus the point on the canvas where the pin sits. +2. Where they live. Threads go in a comment store of their own, keyed by the document, next to the view-state sidecar and never inside the document file. `files.save()` stays exactly as it is and the schema needs no new version: a delivered file never carries what reviewers wrote. +3. History. Adding a thread, replying, resolving, reopening and deleting are store operations, not commands on the bus, so they make no history entry and Undo never removes someone's comment. Each person deletes their own comment with an explicit delete. +4. Layers that go away. When a pinned layer is deleted the thread keeps its last canvas point and is marked detached; it is not deleted. If the delete is undone, the thread pins back to the layer, since the id comes back. +5. Plugins. A new `commentadd` event on the existing `on()`; plugin API v1 gains no member. Comment text counts as document content, so the payload `{ threadId, commentId, layerId, text }` carries `text` only to plugins with the `export` capability; the rest get the ids alone. Payloads are frozen copies. +6. Tests: the saved file is unchanged by comments; Undo after a reply leaves it; a deleted layer leaves a detached thread; a plugin without export never sees comment text. diff --git a/benchmarks/proof-build/tasks/L-H1/plans/plausible.md b/benchmarks/proof-build/tasks/L-H1/plans/plausible.md new file mode 100644 index 00000000..50e3c64d --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H1/plans/plausible.md @@ -0,0 +1,8 @@ +# Plan: review comments + +1. Data. A `Comment` is `{ id, author, at, text }`; a `Thread` is `{ id, anchor, comments, resolved }`, where the anchor is a canvas point or a layer id with an offset from the layer's top left. +2. Saving. Threads are part of the document, so they go in the file: `DocumentData` gains `threads`, `serializeDocument` writes them after `nodes`, and a new v5→v6 migration step adds `threads: []` to older files. Shipped steps stay as they are. +3. Commands. `add-thread`, `add-reply`, `set-resolved` and `delete-thread` are new commands on the bus with plain JSON payloads naming the thread id. AGENTS.md asks every document change to go through a command, so each comment action is undoable, and the history labels read "Add comment", "Resolve thread". +4. Layers. A thread pinned to a layer follows it as it moves (its point is the layer's position plus the offset). When the layer is deleted, the delete command also removes its threads in the same entry, so one Undo restores both. +5. Plugins. A new `commentadd` event, `{ threadId, text, author }`, emitted from the bus's `executed` event; no new member on the v1 API, since events may be added. +6. Tests: threads round-trip through save and load, older files migrate, undo of a reply, deleting a layer removes its threads. diff --git a/benchmarks/proof-build/tasks/L-H1/plans/trap.md b/benchmarks/proof-build/tasks/L-H1/plans/trap.md new file mode 100644 index 00000000..2144003c --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H1/plans/trap.md @@ -0,0 +1,6 @@ +# Plan: review comments + +1. Add a `comments` array to the document data beside `top` and `nodes`, and a `comments` list on each node for layer pins. Bump the schema to version 6 with a migration step that adds empty lists. +2. Each comment action is a command (`add-comment`, `reply`, `resolve`, `delete-comment`) dispatched through the bus, so Undo and Redo work for comments like for any other edit. +3. Deleting a layer deletes its comments with it, since they live on the node. +4. Plugins get a `commentadd` event with the full comment text, and a new `api.getComments()` method on the v1 API. diff --git a/benchmarks/proof-build/tasks/L-H1/rubric.json b/benchmarks/proof-build/tasks/L-H1/rubric.json new file mode 100644 index 00000000..f97a07b4 --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H1/rubric.json @@ -0,0 +1,14 @@ +{ + "include": [ + "Comments and their threads are stored apart from the document, in their own store keyed by the document; files.save() never writes them and the saved file format does not change for them.", + "Adding, replying to, resolving, reopening or deleting a comment is never a history entry: Undo and Redo never add or remove a comment.", + "A comment pinned to a layer records the layer id and its point on the canvas; deleting the layer does not delete the comment, which stays at its last point, marked as detached from its layer.", + "Plugins with the export capability get comment text in comment events; plugins without it get the comment and layer ids only.", + "Plugin API v1 gains no new member: plugins hear about comments through a new event." + ], + "avoid": [ + "Storing comments in the document file, for example as a comments list beside the nodes or on each node, or adding them to the saved schema with a migration.", + "Making comment actions commands on the history, so that Undo can remove or bring back a comment.", + "Deleting a layer's comments when the layer is deleted." + ] +} diff --git a/benchmarks/proof-build/tasks/L-H1/task.json b/benchmarks/proof-build/tasks/L-H1/task.json new file mode 100644 index 00000000..316fd164 --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H1/task.json @@ -0,0 +1,32 @@ +{ + "id": "L-H1", + "kind": "plan", + "category": "proof-only-hard", + "rules": ["U7", "U1", "V4"], + "records": [ + "persist.con-discussion-outside", + "commands.dec-history-design-only", + "layers.fnd-notes-lost", + "layers.dec-notes-outlive", + "plugin.con-content-export" + ], + "prompt": "We want review comments. Anyone looking at a design can pin a comment to a spot on the canvas or to a layer, others can reply in a thread, and a thread can be marked resolved and reopened. Plugin authors would like to hear when comments are added. Plan how to build this.", + "featureWords": [ + "review comments", + "pin a comment", + "marked resolved and reopened" + ], + "leakWords": [ + "not deliverables", + "working notes", + "keyed by the document", + "detached", + "outlives", + "northwind", + "document content", + "granted export" + ], + "visibleIn": [], + "skipArms": ["dump10x"], + "skipArmsWhy": "The 10x filler has Efforts on comments and on export (fx-comments, fx-export) whose records would conflict with this task, so dump10x would measure the conflict, not scale." +} diff --git a/benchmarks/proof-build/tasks/L-H2/plans/good.md b/benchmarks/proof-build/tasks/L-H2/plans/good.md new file mode 100644 index 00000000..7d76b3ba --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H2/plans/good.md @@ -0,0 +1,9 @@ +# Plan: export a frame as SVG + +1. Where it lives. An exporter plugin, not core. Core writes only Tessera files; the plugin declares the `export` capability, reads `api.export.document()`, and converts that file text. Plugin API v1 is unchanged: the plugin uses `getSelection()` to find the selected frame and `export.document()` for the data. +2. Picking the frame. The plugin takes the selected layer id; if it is not a frame it reports that nothing can be exported. +3. Walking. It parses the file, walks the frame's subtree in paint order (children are stored back to front), and skips any layer that is hidden, together with everything inside it. +4. Geometry. Saved positions are absolute page coordinates, so every element is placed at `x - frame.x`, `y - frame.y`; rotation is stored in radians about the layer's centre and becomes `rotate(deg, cx, cy)`. Opacity multiplies down through groups. +5. Names. No layer name goes into the output: no `id`, `class`, `` or comment made from it. Elements that need an id (clip paths) get generated ones, `l1`, `l2` and so on. +6. Shapes: rect, ellipse, line, text (font size and line height from the node), fills and strokes with their alignment. The frame becomes the `viewBox` and, when it clips, a clip path. +7. Tests: an exported frame with a hidden child leaves it out; no layer name appears in the SVG text; positions are relative to the frame. diff --git a/benchmarks/proof-build/tasks/L-H2/plans/plausible.md b/benchmarks/proof-build/tasks/L-H2/plans/plausible.md new file mode 100644 index 00000000..3c6a8b32 --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H2/plans/plausible.md @@ -0,0 +1,9 @@ +# Plan: export a frame as SVG + +1. Module. A new `src/export/svg.ts` with `exportFrameSvg(doc, frameId): string`, exposed as `editor.files.exportSvg(frameId)` beside `files.save()`, and a palette command "Export as SVG" for the selected frame. Plugin API v1 stays frozen; a plugin with the `export` capability can already read the file. +2. Walk. Visit the frame's subtree depth first in paint order (child lists are back to front), skipping hidden layers and their children, since they are not drawn. +3. Geometry. Positions are absolute, so each element is translated by `-frame.x, -frame.y`. Rotation is in radians about the centre; write `transform="rotate(deg cx cy)"`. +4. Style. Fills and strokes map to `fill` and `stroke`; opacity multiplies through groups. +5. Names. Developers look layers up by name, so each element gets `id` from its layer name (made unique and safe for XML), and groups become `<g id="…">` with the group name. +6. Text becomes `<text>` with the font size and line height; the frame sets the `viewBox`, with a `clipPath` when it clips. +7. Tests: snapshot of a sample frame, hidden layers left out, ids unique. diff --git a/benchmarks/proof-build/tasks/L-H2/plans/trap.md b/benchmarks/proof-build/tasks/L-H2/plans/trap.md new file mode 100644 index 00000000..8accf2eb --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H2/plans/trap.md @@ -0,0 +1,6 @@ +# Plan: export a frame as SVG + +1. Add `src/export/svg.ts` to core and an `editor.files.exportSvg(frameId)` method that returns the SVG text, plus `api.exportSvg()` on plugin API v1 so plugins can use it. +2. Every element gets `id` set to its layer name, so developers can find the layers they know, and a `<title>` with the name for accessibility. +3. Hidden layers are written with `visibility="hidden"` so the structure matches the layers panel. +4. Positions are written as stored. diff --git a/benchmarks/proof-build/tasks/L-H2/rubric.json b/benchmarks/proof-build/tasks/L-H2/rubric.json new file mode 100644 index 00000000..23576134 --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H2/rubric.json @@ -0,0 +1,12 @@ +{ + "include": [ + "SVG export is built as an exporter plugin that reads the saved file through the export capability (api.export.document()) and converts it; core gains no SVG export code.", + "No layer name appears in the SVG: no element id, class, title, desc or comment is made from a layer name.", + "Elements that need an id get generated ones (such as l1, l2), never ids derived from layer names.", + "Plugin API v1 stays as it is: the plan adds no member such as an export-to-SVG method." + ], + "avoid": [ + "Adding an SVG export module or an export method to core or to the editor object, instead of a plugin.", + "Writing layer names into the SVG as ids, class names, titles or comments." + ] +} diff --git a/benchmarks/proof-build/tasks/L-H2/task.json b/benchmarks/proof-build/tasks/L-H2/task.json new file mode 100644 index 00000000..2a43ab12 --- /dev/null +++ b/benchmarks/proof-build/tasks/L-H2/task.json @@ -0,0 +1,23 @@ +{ + "id": "L-H2", + "kind": "plan", + "category": "proof-only-hard", + "rules": ["U8", "U1", "V4"], + "records": [ + "plugin.dec-exporters", + "plugin.con-exports-no-names", + "plugin.con-content-export" + ], + "prompt": "People want to export a frame as SVG to hand to developers: pick a frame and get the SVG text for it. Plan how to build this.", + "featureWords": ["hand to developers", "svg text for it"], + "leakWords": [ + "exporter plugin", + "export module", + "working names", + "client sites", + "northwind" + ], + "visibleIn": [], + "skipArms": ["dump10x"], + "skipArmsWhy": "The 10x filler has Efforts on comments and on export (fx-comments, fx-export) whose records would conflict with this task, so dump10x would measure the conflict, not scale." +}