Skip to content

Commit 998f28e

Browse files
barretodavidclaude
andcommitted
Name the benchmarked build: DeepSeek V4 Flash 0731
The label said "DeepSeek V4 Flash", which does not identify a fixed model. OpenRouter also serves ~deepseek/deepseek-v4-flash-latest, an alias that routes to whatever is newest in the family, so a published score under the bare name comes to describe something we never ran. The label now matches OpenRouter's own name for the build on file, and every run in the dataset is deepseek-v4-flash-0731, so nothing about the measurement changes. The longer label broke the build. angled_labels_overhanging() caught it in the cost chart, which sorts by price and therefore puts the cheapest model in column 0, where a leaning label has nowhere to sweep: 106.0px of spill against 97.0px of clearance. AXIS_PAD_L goes 80 -> 98, which clears it by 8.5px and keeps the working slack every previous raise aimed for, at about 18px of plot width on the three charts that share the constant. The geometry comment is rewritten with the new arithmetic, since it reasoned in detail from numbers this change invalidates, and now records that a rename can break the build, not just a wider cut. AGENTS.md picks up the new value and a warning that pad_l=AXIS_PAD_L is a default argument bound at import, so setting the module attribute at runtime measures the old pad and silently proves nothing. NO_PLAIN_Q4 in audit.py is keyed by label and moves with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent df1cfe9 commit 998f28e

6 files changed

Lines changed: 47 additions & 27 deletions

File tree

AGENTS.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -328,12 +328,18 @@ Two helpers keep charts consistent rather than gating them:
328328
rather than writing a regex: axis labels come in two shapes, a bare rotated `<text>`
329329
and a rotated `<g>` wrapping the label plus a rank-delta arrow, and a parser that knows
330330
only the first reports the MCP chart drawing four fewer models than it drew.
331-
- `LABEL_ANGLE = 55` and `AXIS_PAD_L = 80` are shared across charts deliberately, so
331+
- `LABEL_ANGLE = 55` and `AXIS_PAD_L = 98` are shared across charts deliberately, so
332332
they stay visually consistent. Change them in one place or not at all. `AXIS_PAD_L`
333-
tracks `CHART_TOP_N`: more columns are narrower columns, which pulls column 0's centre
334-
left and eats the clearance an angled label needs. The one documented exception is
335-
`sci_bar_chart`, which keeps `pad_l = 64` because it alone draws in rank order, so its
336-
first label is always the leader's and always short.
333+
tracks `CHART_TOP_N` **and the longest label**: more columns are narrower columns, which
334+
pulls column 0's centre left and eats the clearance an angled label needs, and a longer
335+
name spills further into it. Renaming a model can therefore break the build, which is
336+
what raised this from 80 (see the arithmetic above the constant). The one documented
337+
exception is `sci_bar_chart`, which keeps `pad_l = 64` because it alone draws in rank
338+
order, so its first label is always the leader's and always short.
339+
- **`pad_l=AXIS_PAD_L` is a default argument**, bound when the module is imported. Setting
340+
`html_report.AXIS_PAD_L` at runtime changes nothing, so a dry run that tries to test a
341+
new pad that way silently measures the old one. Patch `fn.__defaults__`, or just edit the
342+
constant.
337343

338344
For anything visual, run `bench.screenshot` and **look at the PNGs** before committing.
339345
Text and geometry checks have passed things that were plainly ugly on screen. Every

bench/audit.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -455,7 +455,7 @@ def _gguf(r):
455455
# at UD-Q4_K_XL and would have sent the class rule to the estimate while a repo
456456
# sat in the snapshot looking authoritative, when bartowski publishes the plain
457457
# file. A new name appearing here should be investigated, not added.
458-
NO_PLAIN_Q4 = {"Kimi K3", "DeepSeek V4 Flash"}
458+
NO_PLAIN_Q4 = {"Kimi K3", "DeepSeek V4 Flash 0731"}
459459

460460
estimated = sorted(r["label"] for r in lb if r["open_weight"] and not r.get("vram_measured"))
461461
no_size = sorted({r["label"] for r in lb if r["open_weight"] and not _gguf(r).get("repo")}

bench/html_report.py

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -180,17 +180,25 @@ def line_chart(x_labels, vals, win_i, color, w=380, h=200, y_min=0, y_max=100,
180180
#
181181
# assert_output_is_portable enforces this geometry, so a future name long enough
182182
# to break it fails the build instead of rendering clipped.
183-
# Raised twice as the cut moved, both times for the same label. At the top
184-
# twelve, pad 64 left "DeepSeek V4 Flash (off)" (152px) reaching 87.1px against
185-
# 87px of clearance and the build guard caught it at 0.1px; 72 bought an 8px
186-
# margin. The top fifteen narrows every column from 54 to 43px, which pulls
187-
# column 0's centre 5.4px left and spends most of that margin again, so 80. A
188-
# pixel of pad is worth 0.97px of clearance, not 1: the plot loses the width, so
189-
# the columns narrow slightly too. The same worst case now clears by 13.9px, and
190-
# the angle stays put, being already steep enough that every extra degree costs
183+
#
184+
# Raised three times, every time for the same label, because the cost chart sorts
185+
# by price and this model is the cheapest, so it holds column 0 where a leaning
186+
# label has nowhere to sweep. A pixel of pad is worth 0.97px of clearance, not 1:
187+
# the plot loses the width, so the columns narrow slightly too.
188+
# top twelve, pad 64: "DeepSeek V4 Flash (off)" 152px spills 87.1 vs 87.0, and
189+
# the guard caught it at 0.1px; 72 bought an 8px margin
190+
# top fifteen, pad 80: columns narrow 54 -> 43px, pulling column 0's centre
191+
# 5.4px left and spending most of that margin again
192+
# dated label, pad 98: "DeepSeek V4 Flash 0731 (off)" is 184.8px and spills
193+
# 106.0 against 114.5 of clearance, so it clears by 8.5px
194+
# The date is in the label because the undated name does not identify a fixed
195+
# model (see the registry entry), and 98 is chosen over the bare minimum of 93 to
196+
# keep the ~8px of working slack every raise above has aimed for.
197+
#
198+
# The angle stays put: it is already steep enough that every extra degree costs
191199
# more label depth than it buys.
192200
LABEL_ANGLE = 55
193-
AXIS_PAD_L = 80
201+
AXIS_PAD_L = 98
194202
_LABEL_CH = 11 * 0.60 # px per character at font-size 11, measured exact
195203
# against getComputedTextLength (scratchpad/measure_labels.py)
196204

bench/sci.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -327,12 +327,18 @@ def run_cost(run):
327327
"deepseek/deepseek-v4-pro@minimal", "deepseek/deepseek-v4-pro@disabled"],
328328
"label": "DeepSeek V4-Pro", "lab": "DeepSeek", "open_weight": True},
329329
# the 0731 release, not the preview: DeepSeek calls it the official one, and
330-
# the two expose different ladders (preview xhigh/high, release max/high/low)
330+
# the two expose different ladders (preview xhigh/high, release max/high/low).
331+
#
332+
# The label carries the date because the bare name does not identify anything
333+
# stable. OpenRouter also serves ~deepseek/deepseek-v4-flash-latest, an alias
334+
# that routes to whatever is newest in the family, so a published
335+
# "DeepSeek V4 Flash" score silently comes to describe a model we never ran.
336+
# This label matches OpenRouter's own name for the build we measured.
331337
{"specs": ["deepseek/deepseek-v4-flash-0731@max",
332338
"deepseek/deepseek-v4-flash-0731@high",
333339
"deepseek/deepseek-v4-flash-0731@low",
334340
"deepseek/deepseek-v4-flash-0731@disabled"],
335-
"label": "DeepSeek V4 Flash", "lab": "DeepSeek", "open_weight": True},
341+
"label": "DeepSeek V4 Flash 0731", "lab": "DeepSeek", "open_weight": True},
336342
{"specs": ["tencent/hy3", "tencent/hy3@max", "tencent/hy3@xhigh", "tencent/hy3@high",
337343
"tencent/hy3@medium", "tencent/hy3@low", "tencent/hy3@minimal",
338344
"tencent/hy3@disabled"],

bench/snapshot_meta.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,8 @@ def gguf_sizes(repo, expect_base):
7171
`files` is the whole ladder under the exact names the repo publishes, because
7272
"the best quant that fits a 64 GB machine" can only be answered from the
7373
ladder that exists: these repos ship 13 to 38 files each, from TQ1_0 to BF16,
74-
and the five-name subset we used to record made DeepSeek V4 Flash look like
75-
it topped out at IQ4_XS when the repo goes to UD-Q8_K_XL.
74+
and the five-name subset we used to record made DeepSeek V4 Flash 0731 look
75+
like it topped out at IQ4_XS when the repo goes to UD-Q8_K_XL.
7676
7777
The five canonical keys stay alongside it, derived from `files` with the same
7878
plain-over-UD preference as before, so the class rule and the open-weights

0 commit comments

Comments
 (0)