A sanitized engineering case study covering FSDP-QLoRA on 4×RTX 4090, controlled model evaluation, regression diagnosis and vLLM serving. The public repository includes source-derived implementation examples, executable tests and a CPU-only evaluation demo.
| Capability | Inspectable evidence | Scope |
|---|---|---|
| FSDP-QLoRA integration | Quantized loading and PEFT wrapping | FSDP1 integration excerpt, not a complete training entrypoint |
| Multi-GPU debugging | Per-rank sharding checks | Storage, wrappers, shard-ratio diagnostics |
| Dataset and SFT correctness | Prompt isolation, assistant-only labels | Exact prompt controls, masks and truncation checks |
| Blinded evaluation integrity | Bundle and review validation | Complete coverage, identity binding and assessment consistency |
| Evaluation statistics | Paired cluster metrics | Paired effects, cluster intervals, sign-flip tests and Holm adjustment |
| Post-training decisions | Tradeoff analysis | Separate quality signals; decide the next experiment without overclaiming |
| Recorded outcomes | Training, evaluation, serving | Historical snapshots and explicitly partial provenance |
Python 3.11 or newer; no GPU, model download, credentials or paid services:
python scripts/demo_evaluation.py
python -m unittest discover -s tests -v
python scripts/validate_public_repo.pyThe demo builds synthetic candidates, checks a frozen blind-review bundle, validates hand-authored reviews and calculates paired statistics. It intentionally demonstrates a correctness/completeness tradeoff and labels every output as synthetic, not a measured model result. These CPU checks do not execute the optional GPU integration functions. See evaluation contracts and all implementation boundaries.
Qwen3.8-27B · 4×RTX 4090 · NF4 FSDP-QLoRA · 9,571 training samples · vLLM serving.
| Item | Recorded snapshot |
|---|---|
| LoRA rank / alpha | 16 / 32 |
| Sequence length / effective batch | 2,048 / 32 |
| Epochs / learning rate | 2 / 4e-5 |
| Train / validation / held-out test | 9,571 / 202 / 218 |
| Runtime | Approximately 6h 42m |
| Previously reported trainable parameters | Approximately 116.7M |
The corresponding private record has partial provenance, including a dirty worktree and unknown fields. Training completion is not proof of quality improvement. A later repair-training iteration is described separately in the training summary; its settings and hardware evidence are not merged into this snapshot. The public code is an adaptation, not byte-identical historical execution code.
The project connects data and label checks, constrained training, controlled evaluation, failure diagnosis and serving benchmarks. The goal is to identify what improved, what regressed and what engineering change is justified next, rather than treating a saved adapter or lower training loss as the final deliverable.
The historical broad diagnostic describes 1,000 cases and 2,000 candidate reviews across no-RAG, oracle-RAG and retrieved-RAG. These separate model behavior, evidence use and retrieval effects; the counts are not an accuracy claim.
A later, distinct Decision v3 generation snapshot covers 240 cases × 3 variants = 720 candidate outputs, retaining three truncated outputs. Its main comparison is new SFT versus previous SFT; comparison against pretrained Base is auxiliary. Old regression questions, a fresh decision set and a final confirmation set have different roles. These workloads are not added together as independent evidence.
The new public integrity module and statistics module make selected evaluation controls executable. They are reduced-schema adaptations, not the complete private benchmark. Separate assessment files do not automatically establish independent judges; controlled evidence tasks do not establish production RAG quality. See evaluation details.
The historical vLLM summary reports tensor-parallel serving on four GPUs, concurrency 1/4/8/16/32, and a 300-second sustained workload at concurrency 16. It records zero observed request errors, approximately 21.77 GiB peak memory per GPU, and approximately 334 output tokens/s for the Base sustained run.
Those observations describe the recorded workload, not an SLA. Base and SFT had different output-length distributions, so the throughput difference is not isolated Dynamic-LoRA overhead. A matched fixed-output comparison is needed for that claim. See serving details.
Scoped LoRA/QLoRA implementation and diagnostics; dataset and label audits; frozen Base/SFT evaluation; blind-review quality controls; paired regression analysis; retrieval diagnosis; multi-GPU vLLM capacity testing; and evidence-based recommendations for the next post-training experiment.
This case study does not claim completed DPO/PPO/GRPO training, human-expert release approval or a finished browser-agent benchmark environment. The evaluation components are transferable building blocks, with those additional integrations requiring their own work and tests.
Full case study · Evidence index · Source provenance
- Production RAG Agent — retrieval, reranking, citations and evaluation.
- Industrial CV Production Pipeline — segmentation, spatial/event logic and deployment patterns.
No proprietary datasets, private prompts/responses, weights, credentials or private infrastructure identifiers are published. Tests and the demo use explicitly synthetic fixtures. The formal private experiment-export gate is unchanged; no run JSON, raw review bundle or new measured quality score is fabricated for this update. Hashes identify content, not independent execution or reviewer identity. Detailed limitations are consolidated in provenance. Source-derived code retains its MIT notice, without conveying model or dataset redistribution rights.