Skip to content

Commit 2ac8963

Browse files
barretodavidclaude
andcommitted
Record the bracket-inside-its-error-bar rule
The bracket-first protocol says to extend or top up toward the winning edge, which quietly assumes the bracket found a winner. Gemini 3.7 Flash's did not: three tiers within 0.7 of each other at n=26 against half-widths near 5.5. Topping up all three instead of the nominal winner reordered them and killed a monotone one-shot climb that was an artefact of the small sample, for $1.50. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 74ae20f commit 2ac8963

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

AGENTS.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,16 @@ uv run python -m bench.runner --models qwen/qwen3.6-27b@low,qwen/qwen3.6-27b@hig
269269
- **Use `--rep-offset` past the highest existing rep** for a model already in
270270
`main.jsonl`. Rep numbering restarts per output file, so without it the merge
271271
collides on the identity tuple and you get two different runs claiming to be rep 1.
272+
- **When a bracket lands inside its own error bar, top up every tier, not the winner.**
273+
The standing rule is to top up the variant that won, but that assumes the bracket
274+
identified one. Gemini 3.7 Flash's three tiers came back 86.3 / 86.2 / 85.6 at n=26,
275+
a spread of 0.7 against half-widths near 5.5, so "the winner" was a coin flip and
276+
topping it up alone would have let the flip decide what gets published. At n=52 the
277+
order changed — 87.9 / 85.0 / 87.5, with the provider's own default tier now last
278+
rather than in the middle — and the monotone one-shot climb visible at n=26 (65 → 69
279+
→ 77%) turned out not to exist (71 → 65 → 83%). The extra tiers cost $1.50 on that
280+
model. Compare the bracket's spread against the half-widths before deciding, and if
281+
the spread is inside them, treat the whole bracket as the thing to resolve.
272282
- **No merge tool ships here, and a merge has TWO destinations.** Append the sweep file
273283
into **`main.full.jsonl` in full**, and into **`main.jsonl` with `transcript` and
274284
`final_code` removed**. Dedupe both on `(task, model, condition, rep)`, treat an

0 commit comments

Comments
 (0)