Building AI from matrix multiply to deployment — one architecture at a time.
A pure-Python, zero-dependency AI project that progresses from the simplest possible neural network to production-ready micro-models. Each level teaches a different architecture by applying it to English phonics and literacy education — a domain small enough for a micro-model to master, and useful enough to become a real product.
Built by Nathan Maine
Most AI practitioners start at the framework level — PyTorch, TensorFlow, HuggingFace. They fine-tune pre-trained models without understanding what happens beneath the API. This project goes the other direction: start at the matrix multiply and build up.
This isn't academic. It's the foundation of Memoriant's micro-model product line — proving that we control the entire stack from raw linear algebra through deployment on a $5 microcontroller.
Why teach AI to teach reading? Because:
- The domain is finite. 26 letters, ~44 phonemes, a known set of rules. A micro-model can actually master this.
- The smallest AI learning the smallest unit of language. The story writes itself.
- Real product potential. A reading tutor that runs on any device with no internet — classrooms, developing countries, accessibility tools.
- Maps to every deployment target. Letter recognition fits on an ESP32. Sentence generation fits on a phone. The full system fits on an edge GPU.
| Phase | Name | What Happens |
|---|---|---|
| Phase 1 | From Scratch | Build each architecture in pure Python. Zero dependencies. Every line explained. |
| Phase 2 | Optimize | Apply production techniques: quantization, custom tokenizers, efficient training. |
| Phase 3 | Productize | Package for deployment: SDK, API, device-specific builds. |
| Level | Architecture | Teaches | Literacy Target | Deploy Target |
|---|---|---|---|---|
| A: ABCs | MLP (feed-forward) | Forward pass, loss, backprop, gradient descent | Letter recognition, letter→sound | ESP32 (<100KB) |
| B: Phonics | RNN / LSTM | Sequences, memory, hidden state, gates | Sound blending, CVC words, digraphs | Raspberry Pi (<1MB) |
| C: Reader | Transformer | Attention, embeddings, positional encoding | Sentences, comprehension | Mobile (<50MB) |
| D: Unified | Comparison Study | MoE vs Mamba vs BitNet vs Quantized Transformer | All levels, adaptive | ESP32 → Edge GPU |
Each architecture is chosen because it's the genuinely correct tool for that problem size. An MLP for 26-letter classification. An RNN for sequential sound blending. A transformer for sentence-level understanding. No architecture forced where it doesn't belong.
Want to build this yourself from scratch? There's a companion tutorial repository that walks you through building all 4 architectures in a fill-in-the-blanks format, with tests that verify your implementation at each step.
github.com/NathanMaine/smallest-ai-tutorial
What's in the tutorial repo:
- Tutorial 1: MLP from scratch (7 chapters, 75 tests) — matrix math, forward pass, backprop, SGD, letter classifier
- Tutorial 2: LSTM from scratch (6 chapters, 40 tests) — recurrence, vanishing gradients, gates, BPTT, phonics blender
- Tutorial 3: Transformer from scratch (9 chapters, 60 tests) — embeddings, attention, multi-head, layer norm, full architecture
- Tutorial 4: Architecture comparison (98 tests) — MoE, Mamba, BitNet, quantized transformer, benchmark suite
- Bonus content — BitNet to C export, ARM QEMU verification, architecture decision records
The format: each chapter has starter_code/ with function signatures and docstrings, a solution/ folder if you get stuck, and tests that verify your implementation. You learn by doing, not by reading.
273 tests total. Pure Python standard library. MIT licensed.
If you work through it, open an issue or PR. The confusing parts are the parts that need rewriting.
Each file is a chapter. Read them in order.
# Level A: Start here. Build a neural network from nothing.
cd phase1-from-scratch/level-a-abcs/
python 01_math_foundations.py # Matrix math from scratch
python 02_single_neuron.py # Your first neuron
python 03_forward_pass.py # Data flows through
python 04_loss_function.py # How wrong is the model?
python 05_backpropagation.py # The chain rule applied
python 06_training_loop.py # The training cycle
python 07_letter_classifier.py # It recognizes letters!
python 08_phoneme_mapper.py # It maps letters to sounds!Each file runs standalone. Each file teaches one concept. Each file has extensive comments explaining the math and the why.
smallest-ai-built-from-the-ground-up/
├── README.md # You are here
├── ROADMAP.md # Phase timeline and milestones
├── TODO.md # Active tasks
├── CHANGELOG.md # Every change documented
│
├── docs/ # 110% documentation
│ ├── superpowers/specs/ # Design specification
│ ├── architecture/ # Why we chose each architecture (ADRs)
│ ├── math/ # Mathematical foundations in plain English
│ ├── concepts/ # AI concepts explained from zero
│ └── deployment/ # Device-specific deployment guides
│
├── data/ # Training data by level
│ ├── alphabet/ # Level A: letters, phonemes
│ ├── phonics/ # Level B: rules, digraphs, sight words
│ ├── reader/ # Level C: sentences, stories, Q&A
│ └── unified/ # Level D: combined + metadata
│
├── phase1-from-scratch/ # Pure Python, zero dependencies
│ ├── level-a-abcs/ # MLP (8 chapters)
│ ├── level-b-phonics/ # RNN/LSTM (8 chapters)
│ ├── level-c-reader/ # Transformer (11 chapters)
│ └── level-d-unified/ # Comparison study
│
├── phase2-optimize/ # Production techniques
│ ├── tokenizer/ # Phonics-aware tokenizer
│ ├── quantization/ # int8, int4, binary experiments
│ ├── benchmarks/ # Performance measurement
│ └── ablations/ # Systematic studies
│
├── phase3-productize/ # Memoriant product line
│ ├── models/ # Final trained models
│ ├── api/ # Inference API
│ ├── sdk/ # Python SDK
│ └── deploy/ # Per-target deployment
│
├── tests/ # Tests for everything
└── notebooks/ # Jupyter exploration
Nathan Maine is the #1 ranked engineer on OpenAI's Parameter Golf challenge (1.0925 BPB, beating 400+ competitors). He built a 34.4M parameter model that compresses to under 16MB and trains in under 10 minutes on 8xH100 GPUs. The techniques from that competition — custom tokenizers, quantization, SLOT optimization, Muon optimizer — directly feed Phase 2 of this project.
Memoriant is an AI Factory that builds domain-specific models. The product line spans:
- CMMC Expert — Defense compliance AI (revenue-ready)
- Industrial Benchmark — AI evaluation framework (published on HuggingFace)
- HDP — Conversational AI with personality reproduction (patent pending)
- Janus — Cryptographic supply-chain security (4 patents pending)
- Smallest AI — This project. Micro-model R&D platform.
Six patents pending. 137+ claims filed. Models from 16MB to 72B+ parameters.
This design was validated against the Brain Trust knowledge base (346K+ chunks from 34+ AI/ML experts). Architecture choices confirmed by sentdex, Sebastian Raschka, Trelis Research, and others. See design spec for full validation details.
This project follows a 110% documentation standard:
- Every Python file has a module docstring explaining what, why, and how
- Every function has a docstring explaining the math
- Every architecture choice has a decision record
- Every experiment is logged with hypothesis, method, result, conclusion
- Every directory has a README
- Mathematical foundations are explained in plain English before equations
The goal: read this project top-to-bottom and go from zero AI knowledge to deploying a working neural network on real hardware.
TBD
- Design Specification
- Roadmap
- Changelog
- Parameter Golf Reference (competition records)
- Memoriant