Skip to content

Commit bd0f6b7

Browse files
barretodavidclaude
andcommitted
Record the MCP-on-winner-only rule
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 08ffc79 commit bd0f6b7

1 file changed

Lines changed: 9 additions & 0 deletions

File tree

AGENTS.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,6 +325,15 @@ uv run python -m bench.runner --models qwen/qwen3.6-27b@low,qwen/qwen3.6-27b@hig
325325
(the runner treats an error row as incomplete, so pointing it at the same `--out` retries
326326
exactly those), then count runs per task before merging. The fix is cheap; noticing is the
327327
hard part, since the batch reports a clean 100% solve rate on what survived.
328+
- **The MCP condition is benchmarked on ONE tier: the baseline winner.** Settle the
329+
baseline bracket first (top-ups included), then run `mcp` for that tier alone. Sweeping
330+
the documentation condition across the whole ladder multiplies the slowest, most
331+
contention-sensitive batch by the ladder size for data the report does not need.
332+
Stated 2026-08-25, after GLM 5.3's mcp sweep launched across all three tiers on the
333+
strength of the multi-tier mcp rows already in main.jsonl; those rows predate the rule
334+
and stay as recorded, so do not infer the protocol from the dataset's shape. That
335+
sweep was 57 of 78 runs in when caught and was allowed to finish rather than strand
336+
the spend.
328337
- **Lower it for MCP-condition batches**, 6 to 8: documentation lookups queue on a single
329338
local backend, and unlike retry backoff that wait **is** counted, because `load_runs()`
330339
applies the budget to `llm_time_s + assist_time_s`. A crowded backend manufactures both

0 commit comments

Comments
 (0)