sprint-8: ADR-012 -- sweeps run with cudnn_benchmark false

C8.1b established that disabling cudnn.benchmark alone gives bit-identical
runs; permitting nondeterministic algorithms changed nothing for this workload.
The operational consequence had not been stated: if the noise can be removed
there is no reason to keep estimating it. Ablation sweeps now set
repro.cudnn_benchmark: false, so run-to-run noise at fixed seed is zero and any
difference between two configs is the effect plus seed variance only.

That is the fix for the problem Step 0 opened and Sprint 6 could not close --
whether Mixup or RandAugment beats plain Cutout was 4.5 sigma under one noise
estimate and 1.8 under the other, and the sprint could not say which.

Cost measured, not assumed, in two interleaved blocks per ADR-011:

  cudnn_benchmark: true  (default)  13.99 s/epoch    --
  cudnn_benchmark: false            14.07 s/epoch  +0.6%
  deterministic: true               15.93 s/epoch  +13.8%

+0.6% is indistinguishable from free and is written that way: the block-to-
block spread within the default mode alone was 13.01 to 14.98, wider than the
gap being measured.

The earlier ~25% determinism figure is withdrawn from README, PRD, the sprint 8
spec and EXPERIMENTS.md. It compared runs from different sessions on a drifting
thermal state -- exactly the confound this measurement was built to avoid, and
exactly what ADR-011 exists to name. It survives only in the paragraph that
withdraws it.

ADR-011 is retained; its motivation narrows to keeping timing columns and
thermal exposure fair, since it no longer has to rescue accuracy comparisons.
Sprint 6's sigma ambiguity stands as recorded and is not retro-fitted.

Author: Uditp11

PRD.md

@@ -242,7 +242,7 @@ Bands are estimates from the literature, not promises. If a stage underperforms
 | Risk | Impact | Mitigation |
 |---|---|---|
 | Accuracy plateaus below 94% | Primary metric misses | Sprint 5 has a debug checklist: verify normalization stats, LR range test, confirm no val leakage, check wd is not applied to BN/bias |
-| Bit-exact reproducibility unattainable on GPU | M2 misses | Measured in C8.1: `deterministic=true` **is** bit-exact here, including across a power-state change mid-run, at ~25% throughput cost. The default fast mode is not, and M2 states its measured spread instead of bounding it |
+| Bit-exact reproducibility unattainable on GPU | M2 misses | Measured in C8.1: `deterministic=true` **is** bit-exact here, including across a power-state change mid-run, at ~14% throughput cost. The default fast mode is not, and M2 states its measured spread instead of bounding it |
 | Test set used for model selection | Inflated, dishonest number | Sprint 1 carves a 5,000-image val split from train. Test set is touched **only** by `eval.py`, only at the end of a stage. Enforced by a test that asserts `Trainer` never receives the test loader |
 | Scope creep into a framework | Never ships | Non-goals in §3 are binding; new ideas go to `docs/BACKLOG.md` |
 | Long runs lost to a crash | Time waste | Checkpoint every epoch with resume from Sprint 3, not later |

README.md

@@ -81,7 +81,7 @@ One testable claim per mode, both measured:
   identical `metrics.jsonl` scalars and identical weights. Three 10-epoch runs of
   `configs/repro_check.yaml` hashed to `866f747125539331`, one of them on battery under CPU
   contention, straddling a power-source change mid-run at 2.1× the wall clock. Machine state does
-  not move the number in this mode. Costs about 25% throughput.
+  not move the number in this mode. Costs about 14% throughput (ADR-012).
 - **Default fast mode — expect ~0.1–0.2 run-to-run at a fixed seed; 0.26 observed.**
   `cudnn.benchmark` selects kernels by timing them, so the reduction order depends on machine state
   that no seed controls. `repro.seed` does not make a fast-mode run reproduce. That is the noise

docs/ADR.md

@@ -206,3 +206,61 @@ non-comparable within a sweep either way — `EXPERIMENTS.md` already says so, a
 not fix thermal drift, it only stops that drift from being aliased onto one config. Sprint 6's own
 table keeps the nested ordering it was run with; its σ ambiguity is recorded as open rather than
 retro-fitted.
+
+---
+
+## ADR-012 — Ablation sweeps run with `cudnn_benchmark: false`
+**Date:** 2026-08-03 · **Sprint:** 8 · **Status:** accepted
+
+**Context.** Sprint 6 spent its most valuable measurement on a question it could not close. Step 0
+measured run-to-run spread of **0.21** across clean runs of one recipe, with two runs at an
+*identical seed and config hash* differing by **0.26**. The four-config ablation table then produced
+its own, incompatible estimate of **0.083**, and `EXPERIMENTS.md` records the disagreement as open
+because neither could be trusted: 0.083 came from back-to-back repeats and describes minutes, 0.209
+rests on two degrees of freedom and one high run. The practical cost was concrete — whether Mixup or
+RandAugment beats plain Cutout is 4.5σ under one estimate and 1.8σ under the other, and the sprint
+could not say which.
+
+C8.1b then localised the cause. `deterministic: true` switches off two things at once, and separating
+them showed that **`cudnn.benchmark = False` alone is sufficient for bit-identical runs** —
+permitting nondeterministic algorithms changed nothing for this workload. The variance was never
+inherent to GPU training here; it was kernel selection by wall-clock timing, and the reduction order
+that follows the kernel.
+
+If the noise can be removed, there is no reason to keep estimating it.
+
+**Decision.** Ablation sweeps set `repro.cudnn_benchmark: false`. Two runs of the same config at the
+same seed are then bit-identical, so run-to-run noise at fixed seed is **zero** and any difference
+between two configs is the effect plus *seed* variance only — cleanly separated, and seed variance is
+the quantity that was actually of interest.
+
+**Cost — measured, not assumed.** Three modes, 10 epochs each, run in two interleaved blocks per
+ADR-011, medians over epochs 4–10:
+
+| Mode | Mean median s/epoch | vs default |
+|---|---|---|
+| `cudnn_benchmark: true` (default) | 13.99 | — |
+| **`cudnn_benchmark: false`** | **14.07** | **+0.6%** |
+| `deterministic: true` | 15.93 | +13.8% |
+
+**+0.6% is indistinguishable from free** and should be read that way rather than as a precise figure:
+the block-to-block spread *within* the default mode was 13.01 to 14.98, wider than the gap being
+measured. The honest claim is that turning benchmark off costs nothing detectable, while full
+determinism costs about 14%. An earlier estimate of ~25% for determinism is superseded — it compared
+runs from different sessions on a drifting thermal state, which is the confound this measurement was
+built to avoid.
+
+**Alternatives.** `deterministic: true` for sweeps (rejected: ~14% for no further benefit, since
+benchmark alone already gives bit-identical runs on this workload — and `use_deterministic_algorithms`
+raises on ops with no deterministic kernel, which is a portability tax for nothing). Keep the default
+and average over more seeds (rejected: spends GPU hours estimating noise that can simply be removed;
+Sprint 6's σ work cost about 2.5 h and still did not settle the question). Hold machine state constant
+across a sweep (rejected: unachievable on a laptop over seven hours, and unnecessary — state cannot
+reach the result once kernel selection is fixed).
+
+**Consequences.** Repeats within a sweep must now **vary the seed** to measure anything; two runs of
+one config at one seed are the same run and tell you nothing. ADR-011's interleaved blocks are
+retained but their motivation narrows: they no longer rescue an accuracy comparison, they keep the
+timing columns and thermal exposure fair. Sprint 6's σ ambiguity stands as recorded — it is history,
+and this entry does not retro-fit it. Configs that need the default's throughput can keep it; the
+recommendation binds sweeps, where comparability is the point, not one-off training runs.

docs/EXPERIMENTS.md

@@ -316,8 +316,11 @@ Two configuration findings worth recording rather than discovering later. First,
 AMP path has a deterministic kernel available, so no `warn_only` escape hatch is needed and none was
 used. Second, `CUBLAS_WORKSPACE_CONFIG=:4096:8` is set inside `seeding.seed_everything` *before* any
 CUDA work, which is what keeps cuBLAS from raising on CUDA ≥ 10.2; setting it later, or from a
-shell, would be fragile. Determinism also cost almost nothing here — 15.1–16.7 s/epoch against fast
-mode's 12.6, about 25%, on a model this small.
+shell, would be fragile. Determinism's throughput cost was first put at ~25% here, from 15.1–16.7 s/epoch against fast
+mode's 12.6. **That figure is superseded and withdrawn:** it compared runs from different sessions on
+a drifting thermal state, which is the confound ADR-011 exists to name. Re-measured in two
+interleaved blocks (ADR-012), `deterministic: true` costs **13.8%**, and turning off
+`cudnn.benchmark` alone costs **0.6% — nothing detectable**.
 
 The reason this matters beyond ticking off M2: it answers the Step 0 confound directly. Step 0 could
 not separate "the Energy Saver state changed the result" from "runs vary by 0.26 anyway", because

docs/sprints/08-hardening-release.md

@@ -19,7 +19,7 @@
 
 Also measure and document the throughput cost of `deterministic=true`. Put the number in the README so the tradeoff is explicit rather than folklore.
 
-**Run 2026-08-02 — passed.** Three 10-epoch runs of `configs/repro_check.yaml` were bit-identical, weights hashing to `866f747125539331`, one of them on battery under CPU contention and straddling a replug mid-run at 2.1× the wall clock. Cost of determinism: ~25% (15.1–16.7 s/epoch against fast mode's 12.6). C8.1b then separated the two things `deterministic=true` switches off, and found disabling `cudnn.benchmark` alone sufficient. See `EXPERIMENTS.md`.
+**Run 2026-08-02 — passed.** Three 10-epoch runs of `configs/repro_check.yaml` were bit-identical, weights hashing to `866f747125539331`, one of them on battery under CPU contention and straddling a replug mid-run at 2.1× the wall clock. Cost of determinism: **13.8%**, re-measured in interleaved blocks (ADR-012); an initial ~25% from cross-session runs is withdrawn. Turning off `cudnn.benchmark` alone costs **0.6%**, which is why ADR-012 makes it the sweep default. C8.1b then separated the two things `deterministic=true` switches off, and found disabling `cudnn.benchmark` alone sufficient. See `EXPERIMENTS.md`.
 
 If the two runs diverge beyond tolerance, work backward: is a `DataLoader` worker unseeded? Is an augmentation using a global RNG instead of the seeded generator? Is a non-deterministic CUDA kernel in play that `use_deterministic_algorithms` didn't catch?