Gemini 3.7 Flash enters fourth, and 3.6 retires

Google shipped gemini-3.7-flash on 2026-08-13 and it was benchmarked the same
day: 234 runs, both conditions, $4.65, zero errors and zero retries. Baseline
winner is @low at 87.9 +/-3.7 (n=52), rank 4 of 26, on 156/156 solved. It is
also the fastest thing in the study by some distance, a 6.2s median answer and
a median suite pass 1.8x quicker than Opus 5's, at about a cent a task.

Pinned --provider-order google-ai-studio. Gemini thought signatures only
validate on the endpoint that issued them, and AI Studio was both the faster
half at probe time and the one billing undiscounted; Vertex carries a 50%
launch discount, and pricing a permanent row against a promotion is the defect
the flex-tier rule already covers. Least squares over the runs implies exactly
$0.75/$3.75, so the pin landed where intended.

The probe found two real levels, not four. @disabled is a 400 (reasoning is
mandatory), and @xHigh and @max are unadvertised, accepted, and sit inside
@high's band: 2.7k median first-call output at low against 8.9k/7.1k/9.2k at
high/xhigh/max. Neither unadvertised tier was swept, as with Grok 4.6's @max.

Two findings worth the space. The dial is the mirror image of Grok 4.6's: the
one-shot rate climbs just as honestly, 71% to 83%, but it costs 2.8x the money
and 2.6x the wait, so the index hands back nothing and the three tiers finish
within 2.9 points. And the MCP cells at @low and @Medium are the cleanest noise
measurement in the dataset -- zero tool calls across 52 runs, so they differ
from baseline only by an unused tool definition in the prompt, and they still
moved -5.4 and +3.4. That is the n=26 error bar with the mechanism held at
zero, and it is the scale at which every small lift here should be read.

Gemini 3.6 Flash is deprecated, superseded. Unlike the Grok 4.5 retirement
yesterday this pair is not a tie: 14.7 points on a combined half-width of 8.6,
3.3x cheaper and 4.7x faster. Its 299 runs stay in main.jsonl as the trail.

Also fixes a stale price. Google halved 3.6 Flash after the 2026-08-01 snapshot
-- found while pricing 3.7, since both list at $0.75/$3.75 on the same endpoint
and scoring one against the other's old price would have made the new model
look twice as cheap as it is. All four components moved by exactly 0.5, so it
is expressible; 3.6 was 71.3 and is 73.2 at the corrected price. The FAQ's
"Opus has the fastest median pass" is retired with a check that now asserts the
ordering the sentence claims.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: barretodavid

README.md

@@ -3,7 +3,7 @@
 How good is each frontier LLM at writing Starknet smart contracts on its own, and which
 models get better when you hand them a Cairo documentation tool?
 
-This repository is the benchmark behind both answers: 26 models from 13 labs, **8,327
+This repository is the benchmark behind both answers: 26 models from 13 labs, **8,262
 agentic runs** on 13 hand-written contract tasks, graded by 106 hidden `snforge` tests.
 Every model ran the suite both ways, alone and with the
 [Cairo Coder](https://www.cairo-coder.com/) MCP documentation tool.
@@ -26,14 +26,19 @@ documentation tool:
 | 1 | Opus 5 | low | 92.0 ±0.7 | 100% |
 | 2 | Grok 4.6 | xhigh | 90.3 ±3.2 | 87% |
 | 3 | Fable 5 | xhigh | 89.3 ±1.2 | 100% |
-| 4 | Sonnet 5 | high | 83.4 ±4.3 | 67% |
-| 5 | Kimi K3 | low | 83.2 ±4.2 | 71% |
+| 4 | Gemini 3.7 Flash | low | 87.9 ±3.7 | 71% |
+| 5 | Sonnet 5 | high | 83.4 ±4.3 | 67% |
 
 Intervals are bootstrapped over each model's runs (1,000 resamples), and most of this
-table is a tie. Opus 5 separates from Fable 5, Sonnet 5 and Kimi K3, but not from Grok
-4.6, whose wider interval overlaps it; Grok 4.6 overlaps everything below it too.
+table is a tie. Opus 5 separates from Fable 5 and from Sonnet 5, but not from Grok 4.6
+or Gemini 3.7 Flash, whose wider intervals overlap it. Grok 4.6 overlaps everything
+below it, and Gemini 3.7 Flash overlaps everything except the top spot.
 Read overlapping pairs as ties, not an order.
 
+Gemini 3.7 Flash is the one to look at on price. It reaches fourth at about a cent a
+task, on the fastest median answer in the field at 6.2 seconds, while solving every
+task in every rep; its predecessor sat 15 points lower for three times the money.
+
 All 26 models, the difficulty breakdown and the documentation-tool deltas are in the
 report. To print the full leaderboard yourself:
 
@@ -104,8 +109,8 @@ version of it. Per-model deltas are charted in the report.
 
 ## Published data
 
-`results/runs/main.jsonl` holds 9,214 records: the 8,327 analysed runs, 51 that hit a
-transport error and are dropped at load, and 836 belonging to models since retired from
+`results/runs/main.jsonl` holds 9,448 records: the 8,262 analysed runs, 51 that hit a
+transport error and are dropped at load, and 1,135 belonging to models since retired from
 the study, which the report excludes from every figure including its run count. It is
 the only input the report and the audit need:
 

bench/audit.py

@@ -116,9 +116,16 @@ def check(claim, cond, detail=""):
 o, f, k, mi, s5, sol = (by[n] for n in
     ["Opus 5", "Fable 5", "Kimi K3", "MiMo-V2.5-Pro", "Sonnet 5", "GPT-5.6 Sol"])
 check("Opus 100% one-shot", one(C(o["spec"])) == 100, f"{one(C(o['spec'])):.0f}%")
-check("Opus has the fastest median pass of the charted models",
-      min((pass_time(by[r["label"]]["spec"]), r["label"])
-          for r in lb if r["label"] in charted)[1] == "Opus 5")
+# Opus held this until 2026-08-13, when Gemini 3.7 Flash arrived at half its
+# median pass. The card now concedes the point by name, so the check asserts the
+# ordering the sentence claims rather than just who is first.
+_pass_rank = sorted((pass_time(by[r["label"]]["spec"]), r["label"])
+                    for r in lb if r["label"] in charted)
+check("Gemini 3.7 Flash serves the fastest median pass, 1.8x ahead of Opus",
+      _pass_rank[0][1] == "Gemini 3.7 Flash" and _pass_rank[1][1] == "Opus 5"
+      and abs(_pass_rank[1][0] / _pass_rank[0][0] - 1.8) < 0.1,
+      f"{_pass_rank[0][1]} {_pass_rank[0][0]:.1f}s vs {_pass_rank[1][1]} {_pass_rank[1][0]:.1f}s "
+      f"({_pass_rank[1][0]/_pass_rank[0][0]:.2f}x)")
 check("Opus is 2.7 clear of second", abs(o["sci"] - f["sci"] - 2.7) < 0.06, f"{o['sci']-f['sci']:.2f}")
 check("Kimi 71% vs MiMo 40% one-shot",
       round(one(C(k["spec"]))) == 71 and round(one(C(mi["spec"]))) == 40,
@@ -385,6 +392,34 @@ def chart_labels(svg):
       ", ".join(f"{v:.1f}" for v in g46)
       + f"; one-shot {_g46_one[0]:.0f}% -> {_g46_one[1]:.0f}%"
       + f"; all tiers 100% solved: {all(_g46_solved)}")
+# Gemini 3.7 Flash is the counterweight to Grok 4.6 in the same grid: an equally
+# real dial that the index refuses to reward, because the effectiveness it buys
+# is paid for exactly in cost and speed. The negative half is the claim worth
+# pinning, so the spread is asserted as a ceiling, not just the endpoints.
+g37 = {t: sci(C(f"google/gemini-3.7-flash@{t}")) for t in ("low", "medium", "high")}
+_g37_one = {t: one(C(f"google/gemini-3.7-flash@{t}")) for t in ("low", "high")}
+_g37_x = (med_cost("google/gemini-3.7-flash@high") / med_cost("google/gemini-3.7-flash@low"),
+          st.median([x["llm_time_s"] for x in C("google/gemini-3.7-flash@high")])
+          / st.median([x["llm_time_s"] for x in C("google/gemini-3.7-flash@low")]))
+check("Gemini 3.7's dial buys one-shot, 71 to 83%, for 2.8x the money and 2.6x the wait",
+      max(g37.values()) - min(g37.values()) < 3.0
+      and round(_g37_one["low"]) == 71 and round(_g37_one["high"]) == 83
+      and all(all(x["solved"] for x in C(f"google/gemini-3.7-flash@{t}"))
+              for t in ("low", "medium", "high"))
+      and abs(_g37_x[0] - 2.8) < 0.15 and abs(_g37_x[1] - 2.6) < 0.15,
+      ", ".join(f"{t} {v:.1f}" for t, v in g37.items())
+      + f"; spread {max(g37.values()) - min(g37.values()):.1f}"
+      + f"; one-shot {_g37_one['low']:.0f}% -> {_g37_one['high']:.0f}%"
+      + f"; {_g37_x[0]:.2f}x cost, {_g37_x[1]:.2f}x time")
+# The noise-floor datapoint: two MCP cells where the mechanism is provably absent,
+# because the model never called the tool, and the score still moved either way.
+_g37_mcp = {t: C(f"google/gemini-3.7-flash@{t}", "mcp") for t in ("low", "medium")}
+check("Gemini 3.7's low and medium MCP cells move +/-5 with zero lookups",
+      all(sum(x.get("n_assist_calls") or 0 for x in rs) == 0 for rs in _g37_mcp.values())
+      and abs(sci(_g37_mcp["low"]) - g37["low"]) > 3
+      and abs(sci(_g37_mcp["medium"]) - g37["medium"]) > 3,
+      ", ".join(f"{t}: {sum(x.get('n_assist_calls') or 0 for x in rs)} lookups over {len(rs)} runs, "
+                f"delta {sci(rs) - g37[t]:+.1f}" for t, rs in _g37_mcp.items()))
 fable = {t: sci(C(f"anthropic/claude-fable-5@{t}")) for t in ("xhigh", "max")}
 check("Fable's max drops 5.6 below its xhigh plateau",
       abs(fable["xhigh"] - fable["max"] - 5.6) < 0.15,

bench/html_report.py

@@ -963,7 +963,7 @@ def dial_grid(labels):
 <section>
   <h2>The thinking dial</h2>
   <p class="takeaway" style="margin:0 0 10px">Same index, baseline runs only: the top {word(DIAL_TOP_N)} models, each drawn across its own effort ladder. The ring marks <b>the tier the leaderboard scores</b>.</p>
-  <p class="takeaway" style="margin:0 0 10px">Things to look for: <b>Muse Spark 1.2's dial splits in two</b>, minimal through medium landing at 66 to 71 and high and xhigh at 80. <b>All three Anthropic models pay for <code>max</code></b>: Opus 5 peaks at <code>low</code> and gives back 7 points, while Fable 5 and Sonnet 5 hold plateaus that <code>max</code> drops by 5.6 and 10. Kimi K3 is level at off, low and its <code>max</code> default, dipping only at <code>high</code>. <b>Grok 4.6 is the only dial here that climbs the whole way</b>, 84.4 at <code>low</code> to 90.3 at <code>xhigh</code>, and it buys first-try success rather than correctness: 62% to 87% one-shot with every tier already at 100% solved.</p>
+  <p class="takeaway" style="margin:0 0 10px">Things to look for: <b>Muse Spark 1.2's dial splits in two</b>, minimal through medium landing at 66 to 71 and high and xhigh at 80. <b>All three Anthropic models pay for <code>max</code></b>: Opus 5 peaks at <code>low</code> and gives back 7 points, while Fable 5 and Sonnet 5 hold plateaus that <code>max</code> drops by 5.6 and 10. Kimi K3 is level at off, low and its <code>max</code> default, dipping only at <code>high</code>. <b>Grok 4.6 is the only dial here that climbs the whole way</b>, 84.4 at <code>low</code> to 90.3 at <code>xhigh</code>, and it buys first-try success rather than correctness: 62% to 87% one-shot with every tier already at 100% solved. <b>Gemini 3.7 Flash is the same story with the opposite ending</b>: its one-shot rate climbs just as honestly, 71% to 83%, but the ladder costs it 2.8&times; the money and 2.6&times; the wait, so the index hands back nothing and its three tiers finish within 2.9 points of each other.</p>
   {dial_grid(dial_labels)}
   <div class="legend legend-bottom"><span><span class="key" style="background:{SCI_OPEN_COLOR}"></span>open weights</span><span><span class="key" style="background:{SCI_CLOSED_COLOR}"></span>closed weights</span></div>
 </section>"""
@@ -1002,9 +1002,10 @@ def dial_grid(labels):
     # opposite case, number-forward, and use .scorecard.)
     FAQ = [
         ("Why does Opus 5 win?", "100% one-shot",
-         "Every task in every rep solved on the first submission, at the field's fastest "
-         "median pass. Nothing else in the field is perfect on that measure, which is why a "
-         "flagship price tag still leaves it 2.7 points clear."),
+         "Every task in every rep solved on the first submission. Nothing else in the field is "
+         "perfect on that measure, which is why a flagship price tag still leaves it 2.7 points "
+         "clear. It is no longer the quickest, though: Gemini 3.7 Flash serves a median pass "
+         "1.8&times; faster, and solves everything too."),
         ("Kimi K3 over MiMo? MiMo is far cheaper", "71% vs 40% one-shot",
          "This is the index working as intended. MiMo serves a pass 1.4× faster and 6× cheaper, "
          "and still loses 4.3 points, because it delivers broken code first about three runs in "

bench/sci.py

@@ -105,6 +105,18 @@
     "openai/gpt-5.6-luna-pro":  ((1.00,  6.00, 0.10, 1.250), (0.10, 0.60, 0.01, 0.125)),
     # Z.ai cut GLM 5.2 ~10% in the same week, unrelated to OpenAI.
     "z-ai/glm-5.2":             ((0.7966, 2.5036, 0.14794, None), (0.7168, 2.2528, 0.13312, None)),
+    # Google halved Gemini 3.6 Flash some time after the 2026-08-01 snapshot;
+    # found on 2026-08-13 while pricing 3.7 Flash. Every component moved by
+    # exactly 0.5, so it is expressible. The runs themselves confirm the "was"
+    # vector: least squares over the 299 recorded runs implies $1.34/$7.56,
+    # which is $1.50/$7.50 with cache reads pulling the input average down.
+    #
+    # This one matters beyond its own row. Both Gemini entries are priced at the
+    # google-ai-studio standard endpoint, and 3.7 Flash lists at the same
+    # $0.75/$3.75 there, so without the revision the newer model would appear to
+    # cost half what its predecessor does when the two are in fact identical.
+    "google/gemini-3.6-flash":  ((1.50, 7.50, 0.15, 0.08333333333333334),
+                                 (0.75, 3.75, 0.075, 0.04166666666666667)),
 }
 
 
@@ -562,7 +574,52 @@ def run_cost(run):
     {"specs": ["google/gemini-3.6-flash@max", "google/gemini-3.6-flash@xhigh",
                "google/gemini-3.6-flash@high", "google/gemini-3.6-flash@medium",
                "google/gemini-3.6-flash@low", "google/gemini-3.6-flash@minimal"],
-     "label": "Gemini 3.6 Flash", "lab": "Google", "open_weight": False},
+     # deprecated 2026-08-13: superseded by Gemini 3.7 Flash; charts carry one bar
+     # per family's best (David). Unlike the Grok 4.5 retirement the same day,
+     # this pair is not a tie that needed a tiebreak: 87.9 against 73.2 is a gap
+     # of 14.7 on a combined half-width of 8.6, and 3.7 is also 3.3x cheaper per
+     # task and 4.7x faster. Its 299 runs stay in main.jsonl as the audit trail.
+     "label": "Gemini 3.6 Flash", "lab": "Google", "open_weight": False,
+     "deprecated": True},
+    # Gemini 3.7 Flash, released and benchmarked 2026-08-13 (canonical slug
+    # google/gemini-3.7-flash-20260813). Six endpoints, all status 0, but pinned
+    # --provider-order google-ai-studio regardless: Gemini thought signatures only
+    # validate on the endpoint that issued them, and AI Studio was both the faster
+    # half at probe time (144 tok/s p50 against Vertex's 88) and the one billing
+    # the undiscounted list price. Vertex carries a 50% launch discount, and
+    # scoring a permanent row against a promotion is the defect the flex-tier note
+    # above already rules out. The runs confirm the pin landed: least squares over
+    # them implies exactly $0.75/$3.75. 234 runs, 0 errors, 0 retries.
+    #
+    # Probe: @disabled is a 400, "Reasoning is mandatory for this endpoint and
+    # cannot be disabled", so @low is the floor. OpenRouter advertises low/medium/
+    # high and defaults to medium; @xhigh and @max are unadvertised, accepted, and
+    # land INSIDE @high's band. Median first-call output on the two hard probe
+    # tasks is 2.7k at low against 8.9k / 7.1k / 9.2k at high / xhigh / max: a
+    # 3.3x step up from the floor, then 1.30x across the top three. Two real
+    # levels, so neither unadvertised tier was swept, the same call Grok 4.6's
+    # @max got.
+    #
+    # The dial is inert on the index, and the tiers trade one component for
+    # another rather than climbing: SCI 87.9 / 85.0 / 87.5 at low / medium / high,
+    # every pair overlapping, while one-shot goes 71 -> 65 -> 83% and pays for the
+    # top end in cost (77 -> 58) and speed (100 -> 90). Correctness is 100% at all
+    # three, 156/156. Note @medium, the provider default, is the WORST of the
+    # three rather than the middle: at n=26 it read as the midpoint of a monotone
+    # climb, and only the top-up to n=52 broke that. That is why the top-up went
+    # to all three tiers instead of the nominal winner, and it is the argument for
+    # doing so whenever a bracket lands inside its own error bar.
+    #
+    # The MCP cells are the cleanest noise measurement in the dataset. At @low and
+    # @medium the model called assist_with_cairo ZERO times across 52 runs, so
+    # those cells differ from baseline only by an unused tool definition sitting
+    # in the prompt -- and they still moved -5.4 and +3.4. That is the n=26 error
+    # bar with the mechanism held at zero, not behaviour, and it is the scale at
+    # which every small lift in this study should be read. Only @high touches the
+    # tool at all, 0.42 calls/run over 7 of 26 runs, for +0.6.
+    {"specs": ["google/gemini-3.7-flash@low", "google/gemini-3.7-flash@medium",
+               "google/gemini-3.7-flash@high"],
+     "label": "Gemini 3.7 Flash", "lab": "Google", "open_weight": False},
     {"specs": ["openai/gpt-5.6-luna-pro", "openai/gpt-5.6-luna@max",
                "openai/gpt-5.6-luna@xhigh", "openai/gpt-5.6-luna@high",
                "openai/gpt-5.6-luna@medium", "openai/gpt-5.6-luna@low",

results/model_meta.json

@@ -7,12 +7,27 @@
    "context_length": 1048576,
    "hugging_face_id": null,
    "price_per_m": {
-    "input": 1.5,
-    "output": 7.5,
-    "cache_read": 0.15,
-    "cache_write": 0.08333333333333334
+    "input": 0.75,
+    "output": 3.75,
+    "cache_read": 0.075,
+    "cache_write": 0.04166666666666667
    },
-   "pricing_note": null,
+   "pricing_note": "halved by Google after the snapshot date, found 2026-08-13; the pre-cut $1.50/$7.50 the runs were billed at is the was-vector in sci.PRICE_REVISIONS",
+   "type": null,
+   "params_total": null,
+   "params_active": null
+  },
+  "google/gemini-3.7-flash": {
+   "name": "Google: Gemini 3.7 Flash",
+   "context_length": 1048576,
+   "hugging_face_id": null,
+   "price_per_m": {
+    "input": 0.75,
+    "output": 3.75,
+    "cache_read": 0.075,
+    "cache_write": 0.04166666666666667
+   },
+   "pricing_note": "as listed 2026-08-13 on google-ai-studio, the endpoint the runs were pinned to; the model shipped that day and did not exist at the snapshot date. Google's Vertex endpoints carry a launch discount of 50% that this deliberately does not price against, on the same reasoning that keeps the half-price flex tiers out",
    "type": null,
    "params_total": null,
    "params_active": null