GLM 5.3 enters 20th, starred until Z.ai ships the weights

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: barretodavid

README.md

@@ -3,7 +3,7 @@
 How good is each frontier LLM at writing Starknet smart contracts on its own, and which
 models get better when you hand them a Cairo documentation tool?
 
-This repository is the benchmark behind both answers: 26 models from 13 labs, **8,032
+This repository is the benchmark behind both answers: 27 models from 13 labs, **8,199
 agentic runs** on 13 hand-written contract tasks, graded by 106 hidden `snforge` tests.
 Every model ran the suite both ways, alone and with the
 [Cairo Coder](https://www.cairo-coder.com/) MCP documentation tool.
@@ -39,7 +39,7 @@ Gemini 3.7 Flash is the one to look at on price. It reaches fourth at about a ce
 task, on the fastest median answer in the field at 6.2 seconds, while solving every
 task in every rep; its predecessor sat 15 points lower for three times the money.
 
-All 26 models, the difficulty breakdown and the documentation-tool deltas are in the
+All 27 models, the difficulty breakdown and the documentation-tool deltas are in the
 report. To print the full leaderboard yourself:
 
 ```bash
@@ -109,7 +109,7 @@ version of it. Per-model deltas are charted in the report.
 
 ## Published data
 
-`results/runs/main.jsonl` holds 9,650 records: the 8,032 analysed runs, 51 that hit a
+`results/runs/main.jsonl` holds 9,827 records: the 8,199 analysed runs, 61 that hit a
 transport error and are dropped at load, and 1,567 belonging to models since retired from
 the study, which the report excludes from every figure including its run count. It is
 the only input the report and the audit need:

bench/html_report.py

@@ -694,16 +694,21 @@ def build(all_runs):
     starred = lambda rows: [
         dict(r, label=r["label"] + "*") if r.get("weights_pending") else r for r in rows
     ]
-    # The ONE place the report explains the star, by request: the legend of the
-    # leaderboard chart, where a pending model first appears. Every later star
-    # (the Behind-the-score charts, the MCP chart, the head-to-head paragraph,
-    # the models table's `open*` cell) wears the mark unexplained on purpose.
-    # If a pending model ever ranks below the chart cut this renders nothing
-    # while the table still prints `open*`; the audit's exactly-once count then
-    # fails loudly rather than shipping an unexplained star.
-    pending_note = (
-        '<span>* weights promised but not yet released</span>'
-        if any(r.get("weights_pending") for r in chart_rows) else ""
+    # The ONE place the report explains the star, by request: where a pending
+    # model FIRST APPEARS. That is the leaderboard chart legend when the model
+    # ranks inside the cut, and the models-table caption when it does not;
+    # GLM 5.3 (20th at entry, 2026-08-25) was the first to need the second
+    # site, which used to render nothing and fail the audit by design. The
+    # audit still holds the count at exactly one either way, so both sites
+    # rendering (or neither) fails loudly rather than shipping an unexplained
+    # star. Every later star wears the mark unexplained on purpose.
+    PENDING_NOTE_TEXT = "* weights promised but not yet released"
+    pending_in_charts = any(r.get("weights_pending") for r in chart_rows)
+    pending_note = f"<span>{PENDING_NOTE_TEXT}</span>" if pending_in_charts else ""
+    pending_table_note = (
+        f" <b>{PENDING_NOTE_TEXT}.</b>"
+        if any(r.get("weights_pending") for r in sci_rows) and not pending_in_charts
+        else ""
     )
 
     # Chart 2: best-without vs best-with the MCP, per model. Each condition
@@ -1344,7 +1349,7 @@ def h2h_stats(spec):
     models_html = f"""
 <section>
   <h2>The models</h2>
-  <p class="takeaway" style="margin:0 0 10px">Every model tested, including the {word(len(sci_rows) - CHART_TOP_N)} below the chart cut. Use it to <b>shortlist by the constraint you actually have</b>: the index <b>with and without the documentation tool</b>, <b>price</b> (per million tokens, as listed on OpenRouter, {meta["snapshot_date"]}, or at first listing for models published since), <b>speed</b>, and <b>whether the weights are open</b>. Click a header to sort.</p>
+  <p class="takeaway" style="margin:0 0 10px">Every model tested, including the {word(len(sci_rows) - CHART_TOP_N)} below the chart cut. Use it to <b>shortlist by the constraint you actually have</b>: the index <b>with and without the documentation tool</b>, <b>price</b> (per million tokens, as listed on OpenRouter, {meta["snapshot_date"]}, or at first listing for models published since), <b>speed</b>, and <b>whether the weights are open</b>.{pending_table_note} Click a header to sort.</p>
   <div class="tablewrap"><table id="modeltable" class="sortable">
     <tr><th>Model</th><th class="r desc" data-num aria-sort="descending">SCI</th><th class="r" data-num>SCI (MCP)</th><th class="r" data-num>Δ</th><th>Lab</th><th>Weights</th><th class="r" data-num>Context</th><th class="r" data-num>$/M in</th><th class="r" data-num>$/M out</th><th class="r" data-num>Tok/s</th></tr>
     {"".join(model_rows)}

bench/sci.py

@@ -424,6 +424,30 @@ def run_cost(run):
                "z-ai/glm-5.2@medium", "z-ai/glm-5.2@low", "z-ai/glm-5.2@minimal",
                "z-ai/glm-5.2@disabled"],
      "label": "GLM 5.2", "lab": "Z.ai", "open_weight": True},
+    # Benchmarked 2026-08-25, nine days after the 20260816 checkpoint. One
+    # first-party endpoint (Z.AI, fp8, 1M context), status 0 and uptime
+    # 100/100/99.99 across the 5m/30m/1d windows at probe time; pinned
+    # --provider-order z-ai anyway, per the 0813 precedent, so a third party
+    # appearing mid-sweep cannot split the serving stack unrecorded.
+    #
+    # weights_pending: Z.ai launched 5.3 API-first and said in the announcement
+    # it would publish weights about two weeks after launch (~2026-08-28), once
+    # a cybersecurity safety review closes. zai-org has no 5.3 repo as of
+    # 2026-08-25 and OpenRouter's hugging_face_id is null, so the star stays on
+    # until a repo exists. Z.ai describes it as the 5.2 base unchanged with the
+    # gains from post-training alone, which would put the total near 5.2's
+    # measured 753B; params stay null in model_meta until a repo makes them
+    # checkable (press circulates "743B", which matches nothing verifiable).
+    #
+    # Probe 2026-08-25 (one-line function, 3 reps/tier): @disabled REJECTED
+    # ("Reasoning is mandatory for this endpoint"), so @low is the floor. The
+    # advertised ladder is low/high/max with default max, and the unadvertised
+    # tiers are coerced: @medium lands on @low (28 vs 28 median first-call
+    # tokens) and @xhigh inside @high's band (114 vs 76, ranges overlapping).
+    # Three real levels, all swept.
+    {"specs": ["z-ai/glm-5.3@low", "z-ai/glm-5.3@high", "z-ai/glm-5.3@max"],
+     "label": "GLM 5.3", "lab": "Z.ai", "open_weight": True,
+     "weights_pending": True},
     {"specs": ["minimax/minimax-m3@max", "minimax/minimax-m3@xhigh", "minimax/minimax-m3@high",
                "minimax/minimax-m3@medium", "minimax/minimax-m3@low",
                "minimax/minimax-m3@minimal"],

results/model_meta.json

@@ -976,6 +976,21 @@
      "BF16": 54.7
     }
    }
+  },
+  "z-ai/glm-5.3": {
+   "name": "Z.ai: GLM 5.3",
+   "context_length": 1048576,
+   "hugging_face_id": null,
+   "price_per_m": {
+    "input": 1.4,
+    "output": 4.4,
+    "cache_read": 0.26,
+    "cache_write": null
+   },
+   "pricing_note": "as listed 2026-08-25, the week the model appeared; it did not exist at the snapshot date. Weights promised ~2026-08-28 after a safety review; params stay null until a repo makes them checkable",
+   "type": null,
+   "params_total": null,
+   "params_active": null
   }
  },
  "params_sources": "Lab model cards / HF repo metadata, fetched 2026-07-24: MiMo 1.02T/42B and MiniMax 428B/23B from HF READMEs; DeepSeek 1.6T/49B from HF README (safetensors shows 862B due to FP4/FP8 mixed packing); GLM total 753B from HF safetensors, active ~40B third-party consensus (no lab statement); K3 2.8T/104B lab-stated on the model card, published 2026-07-27 as moonshotai/Kimi-K3; Hy3 295B/21B and Qwen 27B dense lab-stated; V4 Flash 0731 304B from its own card and safetensors tensor count, 13B active from the preview card for the identical architecture (43 layers, 256 experts, 6 routed + 1 shared per token). Muse Glimmer 30B 29.78B dense from the HF safetensors index on meta-models/Muse-Glimmer-30B (29,776,626,688 params, no expert keys in config.json), fetched 2026-08-11. DeepSeek V4 Pro 0813 has no published weights repo and DeepSeek's own docs are unreachable from here, so its parameter counts are left null rather than copied from the preview checkpoint."