Qwen3.8 27B enters 22nd, and the docs are worth +21.8

Alibaba shipped qwen/qwen3.8-27b on 2026-08-14, a dense 27B vision-language
model with public weights. 202 merged runs, both conditions, $19.74, zero
provider retries and one transport error that was retried into a clean cell.
Baseline winner is @low at 18.1 +/-4.7 (n=41), rank 22 of 27.

Pinned --provider-order akashml, and not as a preference: two of the model's
three endpoints sat at status -2 for the whole session, so the only healthy one
was also the only one that keeps this row a single bf16 price basis instead of
a blend with two fp8 providers. audit.py now asserts that pin, as it does for
the 2.4T sibling. Serving is the weak point and the entry says so: the runs
decoded at a median 30.6 tok/s against the suite's 84-95.

The probe cut two tiers. @xHigh is advertised but unusable, with all three cells
past the 900s budget including 1735s on h1_component in a single turn and 1096s
on e1_counter, the easiest task in the suite. @disabled is accepted here, unlike
the 2.4T sibling, and was swept as the bracket floor.

The headline finding is the documentation lift: +21.8, from 18.1 to 39.9, with
correctness going 43.6 -> 93.6 at @low. Qwen3.6-27B gained +22.0. Two 27B models
a generation apart landing within 0.2 of each other is the strongest evidence in
the study that the substitution law belongs to the size class and not to one
checkpoint, and the local-inference chart now shows the two bars side by side.
The mechanism is visible twice: the tool raises the solve rate AND shortens runs
(median 864s -> 436s), which drops the over-budget rate from 37% to 11%.

Worth flagging for the record, because it decided the published variant. 37% of
baseline runs exceeded the 900s model-time budget. At RAW solve rates @low and
@Medium tie at 46%; the budget takes three solves from @Medium and one from @low
and produces 43.6 against 37.8. Eight runs across both conditions passed every
hidden test between 913s and 1005s and are scored as failures, and
m1_erc20_capped@medium did it four separate times. Qwen3.6-27B published at 30%
over budget, so this is in line with precedent rather than new, but it is the
clearest case the dataset has for revisiting the budget for small models.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: barretodavid

README.md

@@ -3,7 +3,7 @@
 How good is each frontier LLM at writing Starknet smart contracts on its own, and which
 models get better when you hand them a Cairo documentation tool?
 
-This repository is the benchmark behind both answers: 26 models from 13 labs, **8,262
+This repository is the benchmark behind both answers: 27 models from 13 labs, **8,464
 agentic runs** on 13 hand-written contract tasks, graded by 106 hidden `snforge` tests.
 Every model ran the suite both ways, alone and with the
 [Cairo Coder](https://www.cairo-coder.com/) MCP documentation tool.
@@ -39,7 +39,7 @@ Gemini 3.7 Flash is the one to look at on price. It reaches fourth at about a ce
 task, on the fastest median answer in the field at 6.2 seconds, while solving every
 task in every rep; its predecessor sat 15 points lower for three times the money.
 
-All 26 models, the difficulty breakdown and the documentation-tool deltas are in the
+All 27 models, the difficulty breakdown and the documentation-tool deltas are in the
 report. To print the full leaderboard yourself:
 
 ```bash
@@ -109,7 +109,7 @@ version of it. Per-model deltas are charted in the report.
 
 ## Published data
 
-`results/runs/main.jsonl` holds 9,448 records: the 8,262 analysed runs, 51 that hit a
+`results/runs/main.jsonl` holds 9,650 records: the 8,464 analysed runs, 51 that hit a
 transport error and are dropped at load, and 1,135 belonging to models since retired from
 the study, which the report excludes from every figure including its run count. It is
 the only input the report and the audit need:

bench/audit.py

@@ -420,6 +420,25 @@ def chart_labels(svg):
       and abs(sci(_g37_mcp["medium"]) - g37["medium"]) > 3,
       ", ".join(f"{t}: {sum(x.get('n_assist_calls') or 0 for x in rs)} lookups over {len(rs)} runs, "
                 f"delta {sci(rs) - g37[t]:+.1f}" for t, rs in _g37_mcp.items()))
+# Qwen3.8 27B is the study's second measurement of the substitution law at the 27B
+# size class, and it lands on top of the first: Qwen3.6-27B gained +22.0, this one
+# +21.8, a generation later. The pair is the load-bearing evidence that the law is
+# a property of the size class rather than of one checkpoint, so both halves are
+# asserted -- the lift itself, and the mechanism, which is that documentation makes
+# runs SHORTER and so hands back cells the 900s budget was eating.
+_q27b = sci(C("qwen/qwen3.8-27b@low"))
+_q27m = sci(C("qwen/qwen3.8-27b@low", "mcp"))
+_q27_ob = lambda c: sum(1 for r in runs if r["model"].partition("@")[0] == "qwen/qwen3.8-27b"
+                        and r["condition"] == c and r.get("over_time_budget"))
+_q27_n = lambda c: sum(1 for r in runs if r["model"].partition("@")[0] == "qwen/qwen3.8-27b"
+                       and r["condition"] == c)
+check("Qwen3.8 27B gains +21.8 from the docs, and the budget losses fall with it",
+      abs((_q27m - _q27b) - 21.8) < 0.2
+      and _q27_ob("baseline") / _q27_n("baseline") > 0.30
+      and _q27_ob("mcp") / _q27_n("mcp") < 0.15,
+      f"{_q27b:.1f} -> {_q27m:.1f} ({_q27m - _q27b:+.1f}); over budget "
+      f"{100 * _q27_ob('baseline') / _q27_n('baseline'):.0f}% baseline -> "
+      f"{100 * _q27_ob('mcp') / _q27_n('mcp'):.0f}% mcp")
 fable = {t: sci(C(f"anthropic/claude-fable-5@{t}")) for t in ("xhigh", "max")}
 check("Fable's max drops 5.6 below its xhigh plateau",
       abs(fable["xhigh"] - fable["max"] - 5.6) < 0.15,
@@ -755,6 +774,14 @@ def fisher(a, b, c, d):
       _q38 and not _unpinned,
       f"{len(_q38) - len(_unpinned)}/{len(_q38)} pinned"
       + (f", {len(_unpinned)} not" if _unpinned else ""))
+# Qwen3.8 27B has the same requirement for a sharper reason: two of its three
+# endpoints sat at status -2 through the sweep, so the pin is what makes the row
+# a single bf16 price basis instead of a blend with two fp8 providers.
+_q27 = [r for r in runs if r["model"].partition("@")[0] == "qwen/qwen3.8-27b"]
+_u27 = [r for r in _q27 if not (r.get("llm_opts") or {}).get("provider_order")]
+check("every Qwen3.8 27B run is pinned to one provider",
+      _q27 and not _u27,
+      f"{len(_q27) - len(_u27)}/{len(_q27)} pinned" + (f", {len(_u27)} not" if _u27 else ""))
 # The local archive is where that material belongs, and every merge should append
 # to both files. Compared by RECORD COUNT rather than by identity tuple: the two
 # files disagree on model ids for legacy records (load_runs folds llm_opts effort

bench/sci.py

@@ -491,6 +491,50 @@ def run_cost(run):
     {"specs": ["qwen/qwen3.8-2.4t-a95b@medium", "qwen/qwen3.8-2.4t-a95b@high",
                "qwen/qwen3.8-2.4t-a95b@low"],
      "label": "Qwen3.8 2.4T A95B", "lab": "Alibaba", "open_weight": True},
+    # Qwen3.8 27B, released 2026-08-14, benchmarked 2026-08-16. Dense 27B
+    # vision-language model, weights public at Qwen/Qwen3.8-27B. 202 merged runs,
+    # 0 provider retries, one transport error that was retried into a clean cell.
+    #
+    # Pinned --provider-order akashml. Only three endpoints exist and two of them
+    # (chutes/fp8, io-net/fp8) sat at status -2 for the whole session, so AkashML
+    # was not a preference but the only healthy option. It happens to be the right
+    # one anyway: bf16 against fp8 elsewhere, and $0.45/$3.20 against a cheapest of
+    # $0.40/$3.00, a 7% premium for a single-quant single-price basis. Serving is
+    # the weak point of this row and the comment should say so plainly: p50
+    # throughput drifted 33 -> 25 tok/s across the session and the runs decoded at
+    # a median 30.6 tok/s, against the 84-95 tok/s the rest of the suite sees.
+    #
+    # Probe: @disabled is ACCEPTED here (reasoning.mandatory is false), unlike the
+    # 2.4T sibling. @xhigh is advertised but unusable — all three probe cells blew
+    # the 900s budget, including 1735s on h1_component in a single turn and 1096s
+    # on e1_counter, the easiest task in the suite. It was not swept, and that is a
+    # budget verdict rather than a quality one.
+    #
+    # THE BUDGET IS THE STORY, and it is worth reading carefully before comparing
+    # this row to anything. 37% of baseline runs exceeded the 900s model-time
+    # budget and were flipped to failures. That is not an outlier: Qwen3.6-27B
+    # published at 30%. But it is doing real work here. At raw solve rates @low and
+    # @medium TIE at 46%; after the budget they read 43.6 and 37.8, so the cutoff,
+    # not capability, is what makes @low the published variant. Eight runs across
+    # both conditions solved every hidden test and were scored as failures, all of
+    # them between 913s and 1005s. m1_erc20_capped@medium did it FOUR separate
+    # times and is recorded as 0-for-4.
+    #
+    # The dial is inert and thinking is not what this model is short of: SCI 14.5 /
+    # 18.1 / 16.5 at disabled/low/medium, every pair overlapping, and the one-shot
+    # rate is 0% at every baseline tier. It never delivers working code first try.
+    #
+    # What it IS short of is Cairo knowledge, and the documentation tool supplies
+    # exactly that: +21.8, from 18.1 to 39.9, with correctness going 43.6 -> 93.6
+    # at @low. That reproduces Qwen3.6-27B's +22.0 almost exactly, one full model
+    # generation later, which is the strongest evidence in the study that the
+    # substitution law is a property of the size class and not of one checkpoint.
+    # The mechanism is visible twice over: the tool raises the solve rate AND
+    # shortens runs (median 864s -> 436s at @low), which drops the over-budget rate
+    # from 37% to 11% and hands back runs the budget was eating.
+    {"specs": ["qwen/qwen3.8-27b@low", "qwen/qwen3.8-27b@medium",
+               "qwen/qwen3.8-27b@disabled"],
+     "label": "Qwen3.8 27B", "lab": "Alibaba", "open_weight": True},
     # Full ladder swept 2026-07-25 for the effort-curve section (both conditions).
     {"specs": ["qwen/qwen3.6-27b@max", "qwen/qwen3.6-27b@xhigh", "qwen/qwen3.6-27b@high",
                "qwen/qwen3.6-27b@medium", "qwen/qwen3.6-27b@low",

bench/snapshot_meta.py

@@ -27,6 +27,9 @@
     "qwen/qwen3.6-35b-a3b":    ("unsloth/Qwen3.6-35B-A3B-GGUF", "Qwen/Qwen3.6-35B-A3B"),
     "google/gemma-4-31b-it":   ("unsloth/gemma-4-31B-it-GGUF", "google/gemma-4-31B-it"),
     "qwen/qwen3.6-27b":        ("unsloth/Qwen3.6-27B-GGUF", "Qwen/Qwen3.6-27B"),
+    # Vision-language, so this repo ships an mmproj projector and a sharded BF16;
+    # the loader already drops mmproj and collapses -00001-of-000NN shard sets.
+    "qwen/qwen3.8-27b":        ("unsloth/Qwen3.8-27B-GGUF", "Qwen/Qwen3.8-27B"),
     # bartowski rather than unsloth: unsloth's Muse Glimmer repo publishes no
     # plain Q4_K_M at all (its 4-bit rung is UD-Q4_K_XL), and Q4_K_M is the
     # exact file the local-inference class rule is defined on.

results/model_meta.json

@@ -913,6 +913,56 @@
      "BF16": 4893.2
     }
    }
+  },
+  "qwen/qwen3.8-27b": {
+   "name": "Qwen: Qwen3.8 27B",
+   "context_length": 262144,
+   "hugging_face_id": "Qwen/Qwen3.8-27B",
+   "price_per_m": {
+    "input": 0.45,
+    "output": 3.2,
+    "cache_read": null,
+    "cache_write": null
+   },
+   "pricing_note": "as listed 2026-08-16 on akashml, the bf16 endpoint the runs were pinned to; the model shipped 2026-08-14 and did not exist at the snapshot date. The two fp8 endpoints are cheaper ($0.40/$3.00 on chutes) but both sat at status -2 through the sweep",
+   "type": "dense",
+   "params_total": "27B",
+   "params_active": "27B",
+   "arch": {
+    "layers": 64
+   },
+   "gguf": {
+    "repo": "unsloth/Qwen3.8-27B-GGUF",
+    "IQ4_XS": 15.7,
+    "Q4_K_M": 17.1,
+    "Q6_K": 22.9,
+    "Q8_0": 29.0,
+    "BF16": 54.7,
+    "files": {
+     "UD-IQ2_XXS": 9.0,
+     "UD-IQ2_M": 10.3,
+     "UD-Q2_K_XL": 10.7,
+     "UD-IQ3_XXS": 11.9,
+     "Q3_K_S": 12.6,
+     "UD-Q3_K_XL": 13.4,
+     "Q3_K_M": 13.8,
+     "IQ4_XS": 15.7,
+     "Q4_0": 16.1,
+     "Q4_K_S": 16.1,
+     "IQ4_NL": 16.3,
+     "Q4_K_M": 17.1,
+     "Q4_1": 17.5,
+     "UD-Q4_K_XL": 17.9,
+     "Q5_K_S": 19.3,
+     "Q5_K_M": 19.8,
+     "UD-Q5_K_XL": 20.2,
+     "Q6_K": 22.9,
+     "UD-Q6_K_XL": 25.9,
+     "Q8_0": 29.0,
+     "UD-Q8_K_XL": 31.5,
+     "BF16": 54.7
+    }
+   }
   }
  },
  "params_sources": "Lab model cards / HF repo metadata, fetched 2026-07-24: MiMo 1.02T/42B and MiniMax 428B/23B from HF READMEs; DeepSeek 1.6T/49B from HF README (safetensors shows 862B due to FP4/FP8 mixed packing); GLM total 753B from HF safetensors, active ~40B third-party consensus (no lab statement); K3 2.8T/104B lab-stated on the model card, published 2026-07-27 as moonshotai/Kimi-K3; Hy3 295B/21B and Qwen 27B dense lab-stated; V4 Flash 0731 304B from its own card and safetensors tensor count, 13B active from the preview card for the identical architecture (43 layers, 256 experts, 6 routed + 1 shared per token). Muse Glimmer 30B 29.78B dense from the HF safetensors index on meta-models/Muse-Glimmer-30B (29,776,626,688 params, no expert keys in config.json), fetched 2026-08-11. DeepSeek V4 Pro 0813 has no published weights repo and DeepSeek's own docs are unreachable from here, so its parameter counts are left null rather than copied from the preview checkpoint."