Skip to content

Commit 51a0131

Browse files
committed
docs: hostile-read fixes to the README
P1: 1. The headline stated 96.06 +/- 0.16% -- an n=2 standard deviation, one degree of freedom, in the largest text on the page. ADR-012 refuses a different number on exactly that ground and the sprint-6 analysis calls an n=2 std nearly meaningless, so the repo taught readers to catch this. Now "96.06% test top-1 -- two seeds, 95.94 and 96.17", with both values in the table and a line saying why. Same for validation and top-5. 2. "About 14%" collapsed ADR-012's deliberate 11-17% range to its midpoint. README now carries the range, and ADR-012's number-of-record clause is updated to match -- it named "about 14%", which this change would otherwise have made stale, which is the drift that clause exists to prevent. PRD aligned too. 3. "expect ~0.1-0.2; 0.26 observed" put the observation outside its own stated range, in the Reproducibility section. Now "typically ~0.1-0.2, and 0.26 is the largest seen". 4. Added the CI badge and a Tests and CI section: 123 tests, what they cover, CPU-only in ~90s, the gpu marker and the dataset skips, the 3.10/3.12 matrix -- and the three real defects the pipeline caught. P2: 5. Fast-mode mechanism corrected. It said reduction order depends on machine state; C8.1b showed permitting nondeterministic algorithms changed nothing, so the mechanism is kernel selection, and different kernels produce different floating-point results. 6. "Machine state does not move the number" softened to what three 10-epoch runs on one GPU support. Ablation count fixed: seven further, being the other seven of the eight-config Sprint 5 sweep -- the eighth is the crop-and-flip row in the table. val-test gap given a unit; top-5 matched to the top-1 convention. 7. Opening clause now names the field and says what CIFAR-10 is, for readers who do not already know. Profile link above the License heading. "Known failure modes: Cats." untouched.
1 parent 443b52c commit 51a0131

3 files changed

Lines changed: 62 additions & 20 deletions

File tree

PRD.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,7 @@ Bands are estimates from the literature, not promises. If a stage underperforms
242242
| Risk | Impact | Mitigation |
243243
|---|---|---|
244244
| Accuracy plateaus below 94% | Primary metric misses | Sprint 5 has a debug checklist: verify normalization stats, LR range test, confirm no val leakage, check wd is not applied to BN/bias |
245-
| Bit-exact reproducibility unattainable on GPU | M2 misses | Measured in C8.1: `deterministic=true` **is** bit-exact here, including across a power-state change mid-run, at ~14% throughput cost. The default fast mode is not, and M2 states its measured spread instead of bounding it |
245+
| Bit-exact reproducibility unattainable on GPU | M2 misses | Measured in C8.1: `deterministic=true` **is** bit-exact here, including across a power-state change mid-run, at roughly 11–17% throughput cost. The default fast mode is not, and M2 states its measured spread instead of bounding it |
246246
| Test set used for model selection | Inflated, dishonest number | Sprint 1 carves a 5,000-image val split from train. Test set is touched **only** by `eval.py`, only at the end of a stage. Enforced by a test that asserts `Trainer` never receives the test loader |
247247
| Scope creep into a framework | Never ships | Non-goals in §3 are binding; new ideas go to `docs/BACKLOG.md` |
248248
| Long runs lost to a crash | Time waste | Checkpoint every epoch with resume from Sprint 3, not later |

README.md

Lines changed: 57 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,30 @@
11
# CIFAR-10 from scratch
22

3-
A CIFAR-10 classifier in PyTorch where the ResNet, the LR schedule, the augmentations and the
4-
training loop are all written in this repository, and every recipe decision is backed by a logged
5-
ablation you can re-run.
3+
[![CI](https://github.com/Uditp11/cifar10-from-scratch/actions/workflows/ci.yml/badge.svg)](https://github.com/Uditp11/cifar10-from-scratch/actions/workflows/ci.yml)
64

7-
**96.06 ± 0.16% test top-1**, mean ± std over two seeds. ResNet-18 (11,173,962 parameters), 200
8-
epochs, one laptop GPU. No pretrained weights, no `timm`, no `torchvision.models`, no Lightning —
5+
An image classification project in computer vision: a deep convolutional neural network — ResNet-18 —
6+
implemented from scratch in PyTorch and trained on CIFAR-10, a standard machine learning benchmark of
7+
60,000 32×32 colour images across 10 categories. The residual blocks, the LR schedule, the
8+
augmentations and the training loop are all written in this repository, and every recipe decision is
9+
backed by a logged ablation you can re-run.
10+
11+
**96.06% test top-1** — two seeds, 95.94 and 96.17. ResNet-18 (11,173,962 parameters), 200 epochs,
12+
one laptop GPU. No pretrained weights, no `timm`, no `torchvision.models`, no Lightning —
913
`torchvision` supplies the CIFAR-10 download and `ToTensor`, nothing else.
1014

1115
| | |
1216
|---|---|
13-
| Test top-1 | **96.06 ± 0.16%** (95.94, 96.17) |
14-
| Test top-5 | 99.73 / 99.75% |
15-
| Validation top-1 | 96.65 ± 0.04% |
16-
| val − test gap | 0.59 |
17+
| Test top-1 | **96.06%** (two seeds: 95.94, 96.17) |
18+
| Test top-5 | 99.74% (two seeds: 99.73, 99.75) |
19+
| Validation top-1 | 96.65% (two seeds: 96.62, 96.68) |
20+
| val − test gap | 0.59 points |
1721
| Training time | 43–49 min per run |
1822
| Hardware | RTX 4060 Laptop GPU, torch 2.11.0+cu128 |
1923

24+
Two seeds is one degree of freedom, so these are reported as both values rather than as mean ± std —
25+
a standard deviation from n=2 carries almost no information, and this repository declines to state
26+
one anywhere (see ADR-012, which refuses a different number on the same grounds).
27+
2028
Test accuracy was measured once, after the configuration was chosen and
2129
[written down first](docs/sprints/06-augmentation.md). `Trainer` takes a `(train, val)` 2-tuple, so
2230
it is structurally incapable of seeing the test set; only `scripts/eval.py` ever loads it.
@@ -69,8 +77,9 @@ between those estimates is [recorded as open](docs/EXPERIMENTS.md) rather than r
6977
the convenient one. Mixup and RandAugment finished 0.03 apart — indistinguishable — so the one that
7078
composes more simply was chosen and the reasoning written down.
7179

72-
Nine further ablations covering optimizer, schedule, warmup, weight decay, label smoothing and epoch
73-
budget are in [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md), along with the results that came out
80+
Seven further ablations covering optimizer, schedule, warmup, weight decay, label smoothing and epoch
81+
budget are in [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) — the other seven of the eight-config
82+
Sprint 5 sweep, the eighth being the crop-and-flip row above — along with the results that came out
7483
against expectation and the claims this project has had to withdraw.
7584

7685
## Reproducibility
@@ -80,11 +89,13 @@ One testable claim per mode, both measured:
8089
- **`repro.deterministic: true` — runs reproduce bit-identically.** Not within a tolerance:
8190
identical `metrics.jsonl` scalars and identical weights. Three 10-epoch runs of
8291
`configs/repro_check.yaml` hashed to `866f747125539331`, one of them on battery under CPU
83-
contention, straddling a power-source change mid-run at 2.1× the wall clock. Machine state does
84-
not move the number in this mode. Costs about 14% throughput (ADR-012).
85-
- **Default fast mode — expect ~0.1–0.2 run-to-run at a fixed seed; 0.26 observed.**
86-
`cudnn.benchmark` selects kernels by timing them, so the reduction order depends on machine state
87-
that no seed controls. `repro.seed` does not make a fast-mode run reproduce. That is the noise
92+
contention, straddling a power-source change mid-run at 2.1× the wall clock. Machine state did not
93+
move the number in this mode (three runs; limits in ADR-012). Costs roughly **11–17%** throughput —
94+
a range on one degree of freedom, not a point estimate (ADR-012).
95+
- **Default fast mode — run-to-run spread at a fixed seed is typically ~0.1–0.2, and 0.26 is the
96+
largest seen.** `cudnn.benchmark` times candidate kernels and picks a winner, so it can select a
97+
*different kernel* depending on machine state, and different kernels produce different
98+
floating-point results. `repro.seed` does not make a fast-mode run reproduce. That is the noise
8899
floor the table above is read against, and it is a property of the mode rather than a defect.
89100

90101
## Known failure modes
@@ -130,6 +141,32 @@ lost that way. Two related habits for long unattended runs: stop the machine sle
130141
and note that a run killed outright is rendered `killed` in the table by `make_report.py`, since a
131142
process destroyed by `TerminateProcess` runs no `finally` and cannot label its own exit.
132143

144+
## Tests and CI
145+
146+
```bash
147+
pytest -m "not gpu" -q
148+
```
149+
150+
**123 tests**, mirroring the `src/` tree, covering config validation and inheritance, the data split
151+
and normalization, augmentation correctness, model shapes and parameter counts, the training loop,
152+
checkpoint/resume, and the run manifest. All run on CPU in about 90 seconds; two more are marked
153+
`gpu` and deselected by default, and ten that need the real CIFAR-10 split skip themselves with a
154+
named reason when it isn't present. CI runs the suite plus `ruff check` and `ruff format --check` on
155+
Python 3.10 and 3.12.
156+
157+
The pipeline earned its place by finding three real defects rather than by existing:
158+
159+
- **A job-level hang.** `pytest` inherited the CIFAR-10 download, produced no output for nineteen
160+
minutes, and was killed by the job timeout with nothing naming the cause. The fetch now runs in its
161+
own bounded step.
162+
- **A bad fetch bound.** Each bound was then set from the previous run's measured progress — 5 minutes
163+
reached 26.4%, 20 minutes reached 99.2% — rather than guessed.
164+
- **A test that passed on tolerance rather than correctness.** `test_resume_matches_uninterrupted_run`
165+
compared a 4-epoch config against a 2-epoch one, and both cosine and warmup scale to that field, so
166+
the two arms diverged *before* the resume it claimed to test. It passed on one platform on a ±0.5
167+
tolerance over a noisy val set and failed on the other. Pinned to a constant LR, the resumed arm now
168+
reproduces the straight arm's per-epoch losses exactly.
169+
133170
## Layout
134171

135172
```
@@ -146,6 +183,10 @@ docs/ARCHITECTURE.md the frozen signatures every component was built against
146183
launching module in each worker, so any script building one guards the call behind
147184
`if __name__ == "__main__":`.
148185

186+
---
187+
188+
Built by [Udit Parihar](https://github.com/Uditp11).
189+
149190
## License
150191

151192
[MIT](LICENSE).

docs/ADR.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -271,9 +271,10 @@ hedging both numbers equally — one difference survives the drift and the other
271271
The recommendation is unchanged either way. `benchmark: false` is adopted because it is free and
272272
removes the noise; what determinism costs on top of it does not bear on that.
273273

274-
**Number-of-record:** the README's Reproducibility section carries **"about 14%"** for determinism,
275-
and nothing for `benchmark: false` beyond the fact that sweeps set it. A re-measurement must update
276-
this entry and that section together, or the two will drift apart.
274+
**Number-of-record:** the README's Reproducibility section carries **"roughly 11–17%"** for
275+
determinism — the range, not the midpoint, so that it cannot state more precision than this entry
276+
supports. It carries nothing for `benchmark: false` beyond the fact that sweeps set it. A
277+
re-measurement must update this entry and that section together, or the two will drift apart.
277278

278279
An earlier estimate of ~25% for determinism is superseded — it compared runs from different sessions
279280
on a drifting thermal state, which is the confound this measurement was built to avoid.

0 commit comments

Comments
 (0)