A production-grade, measured study of LLM inference optimization — serving, batching, KV cache, quantization, distillation, speculative decoding, compiled runtimes, and GPU capacity sizing.
Method, not marketing: every result is a delta vs a baseline with its quality tradeoff attached,
produced by a real GPU run and stamped with {GPU, model, precision, sample size, date}. No number
appears here that didn't come from an actual run. See docs/research/ for the
algorithm/decision-logic notes and docs/specs/ for the design.
Scope honesty: this is a rigorous benchmark study on a laptop GPU (RTX 3060, 6 GB, Ampere) plus a cloud appendix — not a live production service. P0–P5 run locally; P6 is sizing math validated by one cloud point; P7 (FP8 / TensorRT-LLM / Triton / 7B) requires a rented Hopper/Ada GPU.
workloads/ → bench/loadgen → swappable engines/ (HF · vLLM · TensorRT-LLM) → bench/metrics →
bench/report (+ quality anchor) → results/<phase>/ (committed reports + charts).
| Phase | Optimization | Result | GPU |
|---|---|---|---|
| P0 | baseline (HF vs vLLM), DeepSeek-R1-Distill-Qwen-1.5B | vLLM 187.7 vs HF 13.7 tok/s = 13.7×; TPOT 15 ms vs 314 ms (20.9×); GSM8K 0.75 = 0.75 lossless | RTX 3060 6GB |
| P1 | continuous batching + KV cache | max_num_seqs 1→16 = 5.8× throughput (71.7→414.9 tok/s), TTFT 57s→0.32s, quality flat; prefix caching −20% TTFT (lossless) | RTX 3060 6GB |
| P2 | INT4 quantization (AWQ/GPTQ) | AWQ/GPTQ int4 = 1.9× throughput + 2.6× KV-cache headroom, quality = fp16 (lossless) | RTX 3060 6GB |
| P3 | knowledge distillation (1.5B→0.5B SeqKD) | eval-gate caught a regression (0.40→0.27, teacher–student capacity-gap); reliable win = 2.1× throughput / ⅓ memory of the smaller model | RTX 3060 6GB |
| P4 | speculative decoding | measured regression (2.5× slower TPOT, lossless) — wrong tool for a 1.5B target on 6GB; spec-decode needs a large target + high acceptance (→ P7) | RTX 3060 6GB |
| P5 | agentic e2e @ SLO (stacked) | INT4-AWQ + prefix caching = 1.7× throughput / 40% lower $/token / lower p95 at equal concurrency & preserved quality (fair baseline; the big 13.7× is vLLM-vs-HF in P0) | RTX 3060 6GB |
| P6 | GPU sizing | unit-tested sizing calculator anchored on measured 3060, validated (KV math reproduces measured capacity; H100 proj ~11k tok/s matches published); projects L40S/A100/H100 fleet size + $/1M-token | calc + measured anchor |
| P7 | FP8 / TensorRT-LLM / Triton (7B) | code + runbook complete; numbers PENDING a cloud run (FP8 needs Hopper/Ada — honest gap) | cloud (L40S/A100) |
| +X | cross-arch: Liquid LFM2 (hybrid) vs transformer | LFM2 = 2.33× less KV/token → 2.36× more concurrency (6 attn + 10 conv vs 28 attn), widens with context; config-derived. Live GPU serving driver-gated (needs CUDA-13 driver) | config analysis |
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e ".[dev]"
.venv/bin/pytest # 20 GPU-free unit tests
.venv/bin/python scripts/run_baseline.py # P0; other phases: scripts/run_*.pyA reproducible, honesty-stamped methodology applied across the inference-optimization surface, with the judgment to tell wins from mis-applications:
- Big honest win: naive HF Transformers → vLLM = 13.7× throughput (P0, conservative — HF never batches); then INT4-AWQ + prefix caching add 1.7× more / 40% lower $/token at equal concurrency and preserved quality (P5). Decomposed and individually measured (P1 batching scaling curve, P2 1.9×+2.6× KV) — not a single inflated multiple against a strawman baseline.
- Honest negatives that show seniority: distillation into a 0.5B regressed quality — caught by the eval-gate (P3); speculative decoding regressed a 1.5B target — recognized as the wrong tool, not shipped (P4). Both with diagnosed root causes + "when it flips."
- Capacity engineering: a unit-tested GPU-sizing calculator, validated against the measured anchor (P6); a cloud appendix (FP8/TensorRT-LLM) delivered as runnable code + runbook with numbers honestly pending (P7).
Every result: stamped report.json + chart + RESULTS.md + a staff-level ADR (context → measured options
→ decision → when it flips → what we skipped). 9 ADRs, 20 GPU-free unit tests, per-phase commits.