Skip to content

CHANGELOG.md

Latest commit

 

History

History
121 lines (100 loc) · 5.15 KB

CHANGELOG.md

File metadata and controls

121 lines (100 loc) · 5.15 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

[0.5.0] - 2026-04-04 — Level D Complete (Phase 1 Done!)

Added

  • Level D: Unified Comparison Study — 8 chapters, 191 tests
  • Chapter 1: Unified phonics dataset (129 examples from all levels)
  • Chapter 2: Mixture of Experts (MoE) with routing from scratch
  • Chapter 3: Mamba/SSM selective state space model from scratch
  • Chapter 4: BitNet with ternary weights and STE training from scratch
  • Chapter 5: Quantized Transformer with int8 quantization from scratch
  • Chapter 6: Benchmark suite measuring accuracy, size, speed, memory
  • Chapter 7: Comparison report with winner analysis and trade-offs
  • Chapter 8: Recommendation matrix mapping devices to architectures
  • ADR-004: Why comparison study (no single winner)

Architecture

  • Four architectures compared on same dataset: MoE, Mamba, BitNet, Quantized Transformer
  • Benchmark framework for fair comparison
  • Device recommendation matrix: ESP32 → Raspberry Pi → Mobile → Edge GPU

Milestone

  • Phase 1 COMPLETE — All 4 levels implemented, 444 total tests passing
  • Level A: MLP (83 tests)
  • Level B: RNN/LSTM (92 tests)
  • Level C: Transformer (78 tests)
  • Level D: Comparison Study (191 tests)

[0.4.0] - 2026-04-04 — Level C Complete

Added

  • Level C: Reader (Transformer) — 11 chapters, 78 tests
  • Chapter 1: Vocabulary + token embeddings from scratch
  • Chapter 2: Sinusoidal positional encoding
  • Chapter 3: Scaled dot-product self-attention with causal masking
  • Chapter 4: Multi-head attention with parallel heads
  • Chapter 5: Position-wise feed-forward network
  • Chapter 6: Layer normalization + residual connections
  • Chapter 7: Complete transformer block (pre-norm architecture)
  • Chapter 8: Full stacked multi-layer transformer
  • Chapter 9: Training with teacher forcing (analytical output-layer gradients)
  • Chapter 10: Autoregressive sentence generation with temperature sampling
  • Chapter 11: Reading comprehension (passage + question → answer)
  • Training data: 100+ sentences, 10 stories, 20+ comprehension Q&A pairs
  • ADR-003: Why Transformer for reading

Architecture

  • Decoder-only transformer with causal masking
  • Analytical gradients for output projection, transformer as feature extractor
  • Under 50MB model size target (mobile/tablet deployment)

[0.3.0] - 2026-04-04 — Level B Complete

Added

  • Level B: Phonics (RNN/LSTM) — 8 chapters, 92 tests
  • Chapter 1: Recurrence — hidden state and memory concept
  • Chapter 2: Vanilla RNN with Backpropagation Through Time (BPTT)
  • Chapter 3: Vanishing gradients demonstration with ASCII visualization
  • Chapter 4: LSTM cell — forget/input/output gates from scratch
  • Chapter 5: Trainable LSTM sequence model with BPTT through gates
  • Chapter 6: Phonics blender — CVC word blending with LSTM
  • Chapter 7: Phonics rule engine — digraphs, silent-e, vowel teams (neuro-symbolic)
  • Chapter 8: Word reader — complete neuro-symbolic pronunciation model
  • Training data: 50 CVC words, 22 digraph words, 10 silent-e words, 12 vowel team words
  • ADR-002: Why RNN/LSTM for phonics
  • Implementation plan for Level B

Architecture

  • Neuro-symbolic approach: rule engine handles known patterns, LSTM handles learned blending
  • Model size under 1MB (Raspberry Pi deployment target)

[0.2.0] - 2026-04-04 — Level A Complete

Added

  • Level A: ABCs (MLP) — 8 chapters, 83 tests
  • Math foundations from scratch (vectors, matrices, dot product)
  • Single neuron with sigmoid and ReLU
  • Layer and Network classes
  • Softmax, cross-entropy, MSE loss functions
  • Backpropagation with numerical gradient verification
  • Training loop with SGD optimizer
  • Letter classifier: 100% accuracy, 3,418 params, ~13.4 KB
  • Phoneme mapper: 100% accuracy on primary phonemes, 3,452 params, ~13.5 KB
  • ADR-001: Why MLP for ABCs
  • Math documentation (linear algebra, backpropagation)
  • Implementation plan for Level A

[0.1.0] - 2026-04-04

Added

  • Project initialized
  • Design specification v1.0 (docs/superpowers/specs/2026-04-04-smallest-ai-from-scratch-design.md)
  • README.md with project overview, structure, and quick start guide
  • ROADMAP.md with full phase/level task breakdown
  • TODO.md with immediate action items
  • CHANGELOG.md (this file)
  • Directory structure: docs/, data/, phase1-from-scratch/, phase2-optimize/, phase3-productize/, tests/, notebooks/
  • Architecture decision: MLP → RNN/LSTM → Transformer → Comparison Study
  • Brain Trust validation of all architecture choices (346K+ chunks, 34+ experts)
  • Phonics tokenizer concept designed (Phase 2 innovation)
  • Deployment target matrix: ESP32 → Raspberry Pi → Mobile → Edge GPU

Design Decisions

  • Level A uses MLP (not transformer) because letter classification is a classification problem
  • Level B uses RNN/LSTM because phonics blending is inherently sequential
  • Level C uses Transformer because sentence understanding requires attention
  • Level D is a comparison study (MoE vs Mamba vs BitNet vs Quantized Transformer) — no single winner assumed
  • Phonics-specific tokenizer treats phonemes as first-class vocabulary tokens
  • 110% documentation standard: every file, function, and decision documented