sprint-8: re-analyse ADR-012's timings paired within block

ADR-012 applied its skepticism to one number and not the other. It declined to
state +0.6% as precise because block-to-block spread swamped it, then stated
13.8% for determinism -- and that spread is ~14% of the mean, the entire size of
the effect being claimed. A reader who checked would have caught it.

The design already solved this and the analysis was not using it. Interleaving
puts all three modes inside one thermal state per block, so the valid comparison
is the paired within-block difference, where the drift cancels. Pooling each
mode to one mean discards exactly what ADR-011 bought.

Paired, vs the default:

  cudnn_benchmark: false   -1.12s (-7.5%)  +1.27s (+9.8%)   signs disagree
  deterministic: true      +1.61s (+10.8%) +2.25s (+17.3%)  both positive

So the two claims rest on different footing. benchmark=false is not resolvable
-- the measurement cannot say which is faster -- which is free at this
resolution and is what the recommendation rests on. Determinism costs about 14%,
directionally established by both blocks agreeing in sign, with a real range of
11-17% on one degree of freedom rather than a precise figure.

The recommendation is unchanged either way, as it should be: what determinism
costs on top of benchmark=false does not bear on adopting benchmark=false.

ADR-012 now names its number-of-record -- the README carries "about 14%" -- so
the entry and the README cannot drift apart. EXPERIMENTS.md and the sprint 8
spec are aligned to the paired result; the pooled 13.8% / 0.6% figures are
withdrawn and survive only in the sentences that withdraw them.

Author: Uditp11

docs/ADR.md

@@ -234,21 +234,49 @@ same seed are then bit-identical, so run-to-run noise at fixed seed is **zero**
 between two configs is the effect plus *seed* variance only — cleanly separated, and seed variance is
 the quantity that was actually of interest.
 
-**Cost — measured, not assumed.** Three modes, 10 epochs each, run in two interleaved blocks per
+**Cost — measured, and analysed paired.** Three modes, 10 epochs each, in two interleaved blocks per
 ADR-011, medians over epochs 4–10:
 
-| Mode | Mean median s/epoch | vs default |
-|---|---|---|
-| `cudnn_benchmark: true` (default) | 13.99 | — |
-| **`cudnn_benchmark: false`** | **14.07** | **+0.6%** |
-| `deterministic: true` | 15.93 | +13.8% |
-
-**+0.6% is indistinguishable from free** and should be read that way rather than as a precise figure:
-the block-to-block spread *within* the default mode was 13.01 to 14.98, wider than the gap being
-measured. The honest claim is that turning benchmark off costs nothing detectable, while full
-determinism costs about 14%. An earlier estimate of ~25% for determinism is superseded — it compared
-runs from different sessions on a drifting thermal state, which is the confound this measurement was
-built to avoid.
+| Block | `benchmark: true` (default) | `benchmark: false` | `deterministic: true` |
+|---|---|---|---|
+| 1 | 14.98 | 13.86 | 16.59 |
+| 2 | 13.01 | 14.28 | 15.26 |
+
+The absolute numbers drift badly: the default mode alone moves 13.01 → 14.98 between blocks, which is
+~14% of its own mean and as large as the effect being looked for. **Pooling each mode to one mean and
+comparing the means throws away exactly what interleaving bought.** Every block contains all three
+modes under one thermal state, so the valid comparison is the paired within-block difference, where
+the drift cancels:
+
+| Paired difference vs default | Block 1 | Block 2 | Verdict |
+|---|---|---|---|
+| `cudnn_benchmark: false` | **−1.12 s (−7.5%)** | **+1.27 s (+9.8%)** | Signs disagree — **not resolvable** |
+| `deterministic: true` | **+1.61 s (+10.8%)** | **+2.25 s (+17.3%)** | Both positive — **directionally established** |
+
+The two claims therefore rest on different footing, and stating them alike would be wrong in both
+directions:
+
+- **`cudnn_benchmark: false` costs nothing detectable.** The paired differences are −7.5% and +9.8%:
+  opposite signs, so this measurement cannot say which is faster, let alone by how much. Free at the
+  available resolution, and that is the whole basis for the decision above.
+- **`deterministic: true` costs about 14%** — the mean of +10.8% and +17.3%. Both blocks agree in
+  sign and the effect is large against each block's own baseline. Two blocks is one degree of
+  freedom, so 14% is a central estimate with a real range of roughly **11–17%**, not a precise figure.
+
+An earlier version of this entry applied its skepticism asymmetrically: it declined to state +0.6% as
+precise because block-to-block spread swamped it, then stated 13.8% for determinism, which the same
+spread undercuts just as much. The paired analysis above resolves that properly rather than by
+hedging both numbers equally — one difference survives the drift and the other does not.
+
+The recommendation is unchanged either way. `benchmark: false` is adopted because it is free and
+removes the noise; what determinism costs on top of it does not bear on that.
+
+**Number-of-record:** the README's Reproducibility section carries **"about 14%"** for determinism,
+and nothing for `benchmark: false` beyond the fact that sweeps set it. A re-measurement must update
+this entry and that section together, or the two will drift apart.
+
+An earlier estimate of ~25% for determinism is superseded — it compared runs from different sessions
+on a drifting thermal state, which is the confound this measurement was built to avoid.
 
 **Alternatives.** `deterministic: true` for sweeps (rejected: ~14% for no further benefit, since
 benchmark alone already gives bit-identical runs on this workload — and `use_deterministic_algorithms`

docs/EXPERIMENTS.md

@@ -319,8 +319,13 @@ CUDA work, which is what keeps cuBLAS from raising on CUDA ≥ 10.2; setting it
 shell, would be fragile. Determinism's throughput cost was first put at ~25% here, from 15.1–16.7 s/epoch against fast
 mode's 12.6. **That figure is superseded and withdrawn:** it compared runs from different sessions on
 a drifting thermal state, which is the confound ADR-011 exists to name. Re-measured in two
-interleaved blocks (ADR-012), `deterministic: true` costs **13.8%**, and turning off
-`cudnn.benchmark` alone costs **0.6% — nothing detectable**.
+interleaved blocks (ADR-012) and compared *paired within block* so the drift cancels,
+`deterministic: true` costs **about 14%** — +10.8% and +17.3% in the two blocks, agreeing in sign,
+so the effect is directionally established with a real range of roughly 11–17% rather than a precise
+figure. Turning off `cudnn.benchmark` alone is **not resolvable**: its paired differences are −7.5%
+and +9.8%, opposite in sign, so the measurement cannot say which is faster. Free at this resolution,
+which is what ADR-012's recommendation rests on. Both were briefly quoted as pooled means (13.8% and
+0.6%); pooling discards the interleaving and those figures are withdrawn.
 
 The reason this matters beyond ticking off M2: it answers the Step 0 confound directly. Step 0 could
 not separate "the Energy Saver state changed the result" from "runs vary by 0.26 anyway", because

docs/sprints/08-hardening-release.md

@@ -19,7 +19,7 @@
 
 Also measure and document the throughput cost of `deterministic=true`. Put the number in the README so the tradeoff is explicit rather than folklore.
 
-**Run 2026-08-02 — passed.** Three 10-epoch runs of `configs/repro_check.yaml` were bit-identical, weights hashing to `866f747125539331`, one of them on battery under CPU contention and straddling a replug mid-run at 2.1× the wall clock. Cost of determinism: **13.8%**, re-measured in interleaved blocks (ADR-012); an initial ~25% from cross-session runs is withdrawn. Turning off `cudnn.benchmark` alone costs **0.6%**, which is why ADR-012 makes it the sweep default. C8.1b then separated the two things `deterministic=true` switches off, and found disabling `cudnn.benchmark` alone sufficient. See `EXPERIMENTS.md`.
+**Run 2026-08-02 — passed.** Three 10-epoch runs of `configs/repro_check.yaml` were bit-identical, weights hashing to `866f747125539331`, one of them on battery under CPU contention and straddling a replug mid-run at 2.1× the wall clock. Cost of determinism: **about 14%** (11–17% across two interleaved blocks, compared paired so thermal drift cancels — ADR-012). An initial ~25% from cross-session runs is withdrawn, as are the pooled 13.8% / 0.6% figures that briefly replaced it. Turning off `cudnn.benchmark` alone is **not resolvable from free**, which is why ADR-012 makes it the sweep default. C8.1b then separated the two things `deterministic=true` switches off, and found disabling `cudnn.benchmark` alone sufficient. See `EXPERIMENTS.md`.
 
 If the two runs diverge beyond tolerance, work backward: is a `DataLoader` worker unseeded? Is an augmentation using a global RNG instead of the seeded generator? Is a non-deterministic CUDA kernel in play that `use_deterministic_algorithms` didn't catch?