Skip to content

Commit 00494c4

Browse files
phanii9phanii9
committed
initial release: MeowLLM v0.1.0
~3.5M parameter decoder-only transformer (RoPE/RMSNorm/SwiGLU/SDPA) trained from scratch to speak as a house cat named Miso. - Slot-based compositional data generator (15 categories, 20K samples) - Strict shared rules module with whole-phrase banned-phrase matching - 5-dimension eval harness (38 held-out prompts + 8 hard negatives) - 68 pytest tests, GitHub Actions CI (Python 3.10/3.11/3.12) - Bundled CPU-trained checkpoint: 84.2% overall pass rate
0 parents  commit 00494c4

40 files changed

Lines changed: 34407 additions & 0 deletions

‎.github/workflows/test.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: tests
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
matrix:
14+
python-version: ["3.10", "3.11", "3.12"]
15+
16+
steps:
17+
- uses: actions/checkout@v4
18+
19+
- name: Set up Python ${{ matrix.python-version }}
20+
uses: actions/setup-python@v5
21+
with:
22+
python-version: ${{ matrix.python-version }}
23+
24+
- name: Cache pip
25+
uses: actions/cache@v4
26+
with:
27+
path: ~/.cache/pip
28+
key: ${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('pyproject.toml') }}
29+
30+
- name: Install dependencies
31+
run: |
32+
python -m pip install --upgrade pip
33+
pip install torch --index-url https://download.pytorch.org/whl/cpu
34+
pip install -e ".[dev]"
35+
36+
- name: Run tests
37+
run: pytest tests/ -v
38+
39+
- name: Run rule smoke test
40+
run: python scripts/test_rules_smoke.py
41+
42+
- name: Verify generator runs
43+
run: |
44+
python -m meow.generate_data --out-dir /tmp/test_data --n 500 --seed 0
45+
test -f /tmp/test_data/train.jsonl
46+
test -f /tmp/test_data/val.jsonl
47+
48+
- name: Verify tokenizer trains
49+
run: |
50+
python -m meow.tokenizer train /tmp/test_data/train.jsonl /tmp/test_data/tokenizer.json
51+
test -f /tmp/test_data/tokenizer.json
52+
53+
- name: Verify smoke training
54+
run: |
55+
python -m meow.train --smoke --max-smoke-steps 20 \
56+
--train-data /tmp/test_data/train.jsonl \
57+
--val-data /tmp/test_data/val.jsonl \
58+
--tokenizer /tmp/test_data/tokenizer.json \
59+
--out-dir /tmp/test_ckpt

‎.gitignore‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
build/
8+
dist/
9+
*.egg-info/
10+
*.egg
11+
.pytest_cache/
12+
.ruff_cache/
13+
.mypy_cache/
14+
.coverage
15+
htmlcov/
16+
17+
# Virtual environments
18+
venv/
19+
env/
20+
.venv/
21+
ENV/
22+
23+
# IDE
24+
.vscode/
25+
.idea/
26+
*.swp
27+
*.swo
28+
.DS_Store
29+
30+
# Jupyter
31+
.ipynb_checkpoints/
32+
33+
# Training artifacts (regenerated locally)
34+
checkpoints/
35+
wandb/
36+
runs/
37+
*.pt
38+
*.bin
39+
*.safetensors
40+
41+
# Model weights from HF are NOT checked in — they're hosted
42+
# Dataset files are intentionally tracked (regenerating takes time)
43+
# and live in data/ — but delete them if you prefer to regenerate:
44+
# data/*.jsonl
45+
# data/tokenizer.json
46+
47+
# Logs
48+
*.log
49+
50+
# OS
51+
Thumbs.db

‎CHANGELOG.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Changelog
2+
3+
All notable changes to MeowLLM will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
---
9+
10+
## [0.1.0] — Initial release
11+
12+
First public release of MeowLLM / Miso.
13+
14+
### Added
15+
16+
#### Model
17+
18+
- ~3.5M parameter decoder-only transformer (`meow/model.py`, 278 lines)
19+
- Modern architecture: RoPE, RMSNorm, SwiGLU, torch SDPA, tied embeddings
20+
- Config: 4 layers, d_model 256, 4 heads, ffn_hidden 640, context 256
21+
- Causal self-attention with flash-attention kernels via SDPA
22+
- Autoregressive generation with temperature, top-k, and EOS handling
23+
24+
#### Data pipeline
25+
26+
- Slot-based compositional template generator (`meow/generate_data.py`)
27+
- 15 categories fully populated with inputs, cores, openers, sensories,
28+
and redirects
29+
- Per-category probability tuning for optional slots
30+
- Optional LLM augmentation path via Anthropic API (disabled by default)
31+
- Robust JSON extraction from LLM responses (handles fences, prose)
32+
- Deduplication on `(input, output)` tuples
33+
- Eval-prompt leakage prevention during generation
34+
- Rejection stats reporting by reason
35+
36+
#### Rules module
37+
38+
- Single source of truth for character validation (`meow/rules.py`)
39+
- Whole-phrase matching for banned assistant phrases (no substring false
40+
positives)
41+
- Asymmetric per-category keyword requirements
42+
- Exempt-category handling for nonsense questions
43+
- Short-response bypass (≤6 words skip keyword checks)
44+
- 40+ banned phrases across AI disclosure, helpfulness, chirpiness, and
45+
refusal-as-assistant categories
46+
47+
#### Tokenizer
48+
49+
- Byte-level BPE tokenizer trained on generated dataset
50+
(`meow/tokenizer.py`)
51+
- Target vocab 2048, typical trained vocab ~1682
52+
- Special tokens: `<pad>`, `<bos>`, `<eos>`, `<user>`, `<miso>`
53+
- Chat format encoding with `output_start` index for loss masking
54+
55+
#### Training
56+
57+
- AdamW optimizer with cosine decay + linear warmup (`meow/train.py`)
58+
- Gradient clipping at 1.0
59+
- Smoke mode for fast pipeline validation (`--smoke`)
60+
- Best and final checkpoint saving
61+
- Training metadata JSON export
62+
63+
#### Dataset class
64+
65+
- PyTorch `MeowDataset` with proper loss masking (`meow/dataset.py`)
66+
- User-turn positions marked with `IGNORE_INDEX = -100`
67+
- Padding to fixed max sequence length
68+
- Collate function for batching
69+
70+
#### Evaluation
71+
72+
- Held-out prompt suite: 38 prompts across 15 categories + hard negatives
73+
(`meow/eval_cases.py`)
74+
- 5-dimension evaluation: lowercase, length, banned phrases, cat framing,
75+
full gate
76+
- Batch evaluation with per-check pass rates
77+
- Top-failure-reasons reporting
78+
79+
#### Inference
80+
81+
- Checkpoint loading with config reconstruction (`meow/inference.py`)
82+
- Single-prompt and interactive modes
83+
- Proper device handling across CPU/GPU
84+
85+
#### Tests
86+
87+
- 68 pytest tests covering:
88+
- Rules module (34 test cases)
89+
- Generator behavior and yield
90+
- Cross-consistency between `CATEGORIES` and `CATEGORY_KEYWORDS`
91+
- Model architecture (shapes, RoPE identity, RMSNorm, ignore_index,
92+
tied embeddings)
93+
- Tokenizer round-trip and chat format
94+
- Dataset loss masking
95+
- Evaluation harness
96+
- Runs in ~7 seconds on CPU
97+
98+
#### Notebooks
99+
100+
- `notebooks/train_meow.ipynb` — one-click Colab training
101+
- `notebooks/chat_with_meow.ipynb` — HF-download-first chat with
102+
training fallback
103+
104+
#### Documentation
105+
106+
- `README.md` — project overview, quick start, architecture, evaluation
107+
- `persona.md` — character bible with hard voice rules
108+
- `CONTRIBUTING.md` — contribution guide
109+
- `docs/getting_started.md` — friendly tutorial walkthrough
110+
- `docs/troubleshooting.md` — common issues and fixes
111+
- `docs/faq.md` — frequently asked questions
112+
- `docs/release.md` — step-by-step release process
113+
- `docs/dataset_card.md` — Hugging Face dataset card (valid YAML front matter)
114+
- `docs/model_card.md` — Hugging Face model card (valid YAML front matter)
115+
- `CITATION.cff` — valid CFF 1.2.0
116+
- Inline docstrings on every public function and class
117+
118+
#### Packaging
119+
120+
- `pyproject.toml` with setuptools backend
121+
- Console scripts: `meow-generate`, `meow-tokenizer`, `meow-train`,
122+
`meow-chat`
123+
- Optional dependency groups: `llm` (Anthropic), `hub` (HuggingFace),
124+
`dev` (pytest, ruff)
125+
- Python 3.10+ support
126+
- `LICENSE` — MIT
127+
128+
#### Infrastructure
129+
130+
- GitHub Actions CI (`.github/workflows/test.yml`) testing Python
131+
3.10, 3.11, 3.12
132+
- CI runs pytest, rules smoke test, generator, tokenizer, and smoke
133+
training to verify end-to-end pipeline
134+
- `scripts/test_rules_smoke.py` — portable rules smoke test (no
135+
hardcoded paths)
136+
- `scripts/upload_to_hf.sh` — one-shot Hugging Face upload for model
137+
and dataset
138+
139+
### Baseline numbers
140+
141+
Pass rates on held-out eval (bundled CPU checkpoint, 2000 steps, val_loss 0.476):
142+
143+
- lowercase: 100.0%
144+
- length: 100.0%
145+
- no banned phrases: 100.0%
146+
- cat framing: 81.6%
147+
- **overall: 84.2%**
148+
149+
A full 10-epoch GPU training run is expected to produce higher numbers still.
150+
Contributors who complete a GPU run are invited to open a PR with their numbers.
151+
152+
### Known limitations
153+
154+
- Only the smoke-trained checkpoint has been validated. Full-training
155+
numbers pending.
156+
- Narrow vocabulary (~1700 BPE tokens).
157+
- Single-turn only.
158+
- English/lowercase only.
159+
- HF upload script untested on real credentials.
160+
- Colab notebook untested on a real Colab instance (cells are
161+
syntactically correct and commands have been validated locally).
162+
163+
---
164+
165+
## [Unreleased]
166+
167+
Things planned for future releases:
168+
169+
- Pretrained weights published to Hugging Face
170+
- Verified full-training eval numbers in the model card
171+
- ONNX export path for browser inference
172+
- Optional multi-turn format (experiment)
173+
- `docs/architecture.md` deep-dive on design decisions

‎CITATION.cff‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
cff-version: 1.2.0
2+
message: "If you use MeowLLM, please cite it as below."
3+
title: "MeowLLM: a tiny character language model that talks like a house cat"
4+
version: 0.1.0
5+
date-released: 2026-04-06
6+
authors:
7+
- family-names: phanii9
8+
given-names: ""
9+
repository-code: "https://github.com/phanii9/MeowLLM"
10+
license: MIT
11+
keywords:
12+
- language-model
13+
- transformer
14+
- character-model
15+
- educational
16+
- tiny-model
17+
abstract: >-
18+
MeowLLM is a ~3.5M parameter decoder-only transformer trained from scratch
19+
to speak in the voice of a single house cat character called Miso.
20+
Built with modern tiny-LM components (RoPE, RMSNorm, SwiGLU, SDPA, tied
21+
embeddings) and a slot-based compositional synthetic data generator with
22+
strict character filtering.

0 commit comments

Comments
 (0)