v0.12.1: Unsloth 2026.9.2 / vLLM 0.28.0 images validated on Qwen3.8-27B and the full cookbook; DPO, GPU-reclaim and API-deploy fixes; training diagnostics

[nielsrolf]

Summary

• Worker images rebuilt and released (v0.12, then v0.12.1). Unsloth 2026.9.2 on Torch 2.10.0+cu128 / Transformers 5.5.0 / TRL 0.24.0; vLLM 0.28.0 on Torch 2.13.0+cu130 / Transformers 5.16.1. Dependencies are resolved together and pip check is enforced. Unused MergeKit/LLM Blender dropped from the Unsloth image. SDK defaults and package version are now v0.12.1 / 0.12.1. Release tags are thin overlays on the GPU-validated candidates, so pip freeze is byte-identical (results/image-freeze-*.txt).
• Qwen/Qwen3.8-27B validated on GPU: SFT ftjob-a14928964512, DPO ftjob-18fa65aeea16, inference inferencejobs-56262ef4e688. v0.11 failed at load.
• Full integration suite run (tests/test_integration.py, Docker build-and-push skipped so it cannot overwrite validated tags; new --skip flag): signup/tokens passed, local worker passed, all 15 cookbook examples passed (SFT on Qwen3.5-0.8B, Qwen3-4B, Gemma 3 4B, OLMo 3 7B, Qwen3.5-35B-A3B, QLoRA Llama 3.3 70B; logprob tracking; sampling callback; token-weighted SFT; Llama 3.1 8B DPO and ORPO; batch inference; Qwen3.6-35B-A3B inference; custom job; API deployment).

Bugs fixed (each found by a real failure in this PR's validation)

1. DPO on Qwen3.5/Qwen3.8 crashed after loading. TRL 0.24 flags model_type=qwen3_5 as a vision model and tokenizes preference rows through processing_class.tokenizer. TextPreferenceDPOTrainer uses text tokenization when given a plain tokenizer. Also: DPO now passes the normalized LR, max_length and eval schedule to DPOConfig.
2. Worker GPU reclaim caused unbounded pod churn. reclaim_gpu treated any GPU-holding PID outside the container as fatal. RunPod hosts commonly expose a host-side process holding ~700 MiB: a single inference job spun up 88 H100 NVL pods in two hours without ever running (~3.45 pod-hours), and 9 of 13 H200 pods did the same during the smoke tests. Foreign holders are now ignored while min_free_fraction is met. Shipped as v0.12.1 because worker code is baked into the images. Follow-up: the cluster manager has no back-off when workers fail before running a job.
3. ow.api.deploy() crashed unless OPENAI_API_KEY was set. TemporaryApi.up() built OpenAI clients with api_key=None. Now reads the key from job params like the async path.
4. Inference CLI: removed the deleted snapshot_download(local_dir_use_symlinks=...) argument. validate.py accepts adamw_torch.

Also included

• Rebuilt dashboard frontend assets for the merged PRs (Fix dashboard option page #73, Fix password reset flow: add missing /reset-password route #46); committed with --no-verify like previous builds.
• Training diagnostics (experiments/training_comparison/): OpenWeights vs Tinker on Qwen3-8B with deterministic data, shared evaluation masks, two seeds, an LR sweep, a 128/128 trainer/collator label audit, an adapter-metadata audit and an alpha-32 control. OW and Tinker match closely at LR 1e-4 and 1e-3; at 1e-5 OW learned much more slowly (test NLL 4.17 vs 0.89). Tinker's adapter uses lora_alpha=32 vs OW's 16; an OW alpha-32 control roughly halves the log-NLL gap but does not close it. Remaining confounds are documented, not resolved. See REPORT.md, STATUS.md, LARGE_MODELS.md (GLM-5.3-scale training needs a sharded backend; the native trainer is single-device).

Validation

• 36 unit tests pass; formatting hooks pass.
• results/image-validation.json has digests, job IDs and per-example integration results; results/integration-worker-accounting.json has pod-hours.
• Cluster image v0.12.1 passes its build-time runtime check but was not redeployed.

Not included

• Feature PRs feat(runpod): allow requesting a minimum host vCPU / RAM for pods #76, Add per-job hf_token/hf_org override for fine-tuning jobs #61, docs: mirror the RunPod API key safety rule into AGENTS.md #77 left open (outside bug-fix scope).

🤖 Generated with Claude Code