v0.12.1: Unsloth 2026.9.2 / vLLM 0.28.0 images validated on Qwen3.8-27B and the full cookbook; DPO, GPU-reclaim and API-deploy fixes; training diagnostics

.dockerignore

@@ -97,3 +97,7 @@ example/_*
 openweights/jobs/unsloth/check.ipynb
 .cache
 .logs/
+
+.venv/
+.git/
+experiments/

DOCKER_README.md

@@ -14,7 +14,10 @@ VERSION=$(python -c "from openweights.images import IMAGE_VERSION; print(IMAGE_V
 
 ## 1. Unsloth Worker Image
 
-Built from `unsloth/unsloth:latest` and kept close to upstream Unsloth.
+Built from `pytorch/pytorch:2.10.0-cuda12.8-cudnn9-runtime` with Unsloth `2026.9.2`.
+The SDK and worker dependencies are resolved together and `pip check` must pass.
+MergeKit and LLM Blender are not used by the built-in jobs and are no longer installed
+in this image; custom jobs that need them should use a separate compatible image.
 
 ```sh
 docker buildx build \
@@ -25,7 +28,9 @@ docker buildx build \
 
 ## 2. vLLM Worker Image
 
-Built from `vllm/vllm-openai:v0.19.1` for a clean `vllm` + `transformers 5.x` stack.
+Built from `vllm/vllm-openai:v0.28.0`, preserving its matching PyTorch/CUDA stack,
+with Transformers `5.16.1` and TRL `0.24.0`. The FastAPI constraint matches vLLM.
+Both dependency consistency and GPU model smoke tests are required before promoting a tag.
 
 ```sh
 docker buildx build \
@@ -80,3 +85,29 @@ together, then rebuild. Checksums are on the
 [release page](https://github.com/bcpierce00/unison/releases). Keep the two images on the same
 version, and prefer whatever Homebrew currently ships, since that is what most clients will
 install.
+
+## Candidate validation and promotion
+
+Build and publish a distinct candidate tag before updating `openweights/images.py`.
+Run Qwen3.8-27B SFT, DPO, and inference against the candidate; the scripts in
+`experiments/training_comparison/` accept explicit image tags. Save resolved package
+versions, image digests, job IDs, worker logs, and generated samples. A successful
+Docker build alone does not establish GPU compatibility.
+
+Promote a validated candidate without re-resolving dependencies: build the release tag
+as an overlay that only refreshes the SDK source and package metadata, then confirm
+`pip freeze` is unchanged.
+
+```sh
+cat > /tmp/Dockerfile.release <<'RELEASE'
+FROM nielsrolf/ow-unsloth:${VERSION}-candidate
+WORKDIR /openweights
+COPY README.md pyproject.toml ./
+COPY openweights openweights
+RUN /opt/venv/bin/python -m pip install --no-cache-dir --no-deps -e . && /opt/venv/bin/python -m pip check
+RELEASE
+docker build -f /tmp/Dockerfile.release -t nielsrolf/ow-unsloth:$VERSION . && docker push nielsrolf/ow-unsloth:$VERSION
+```
+
+The cluster image uses the system `python3` instead of `/opt/venv/bin/python`. Records for
+v0.12 are in `experiments/training_comparison/results/image-validation.json`.

Dockerfile

@@ -30,28 +30,15 @@ COPY README.md .
 COPY pyproject.toml .
 COPY openweights openweights
 COPY entrypoint.sh .
-RUN python3 -m venv --system-site-packages /opt/venv && \
+# spin is a PyTorch source-build helper, not a runtime dependency. Its Click
+# upper bound conflicts with the current Hugging Face CLI.
+RUN python3 -m pip uninstall --break-system-packages -y spin && \
+    python3 -m venv --system-site-packages /opt/venv && \
     /opt/venv/bin/python -m pip install --no-cache-dir --upgrade pip && \
     /opt/venv/bin/python -m pip install --no-cache-dir \
-        "unsloth[cu128-torch2100]==2026.4.6" && \
-    /opt/venv/bin/python -m pip install --no-cache-dir --no-deps -e . && \
-    /opt/venv/bin/python -m pip install --no-cache-dir \
-        PyJWT \
-        cachier \
-        diskcache \
-        fastapi \
-        fire \
-        "httpx[http2]>=0.24.0" \
-        huggingface-hub \
-        openai \
-        python-dotenv \
-        runpod \
-        scp \
-        "supabase==2.15.3" \
-        uvicorn \
-        hf_transfer \
-        "mergekit==0.1.4" \
-        "llm-blender==0.0.2"
+        "unsloth[cu128-torch2100]==2026.9.2" \
+        "huggingface-hub>=1.23,<2" "safetensors>=0.8" hf_transfer -e . && \
+    /opt/venv/bin/python -m pip check
 
 RUN /opt/venv/bin/python - <<'PY'
 import importlib.metadata as metadata

Dockerfile.vllm

@@ -1,18 +1,12 @@
 # syntax=docker/dockerfile:1.7
-FROM pytorch/pytorch:2.10.0-cuda12.8-cudnn9-runtime
+FROM vllm/vllm-openai:v0.28.0
 
 USER root
 
 WORKDIR /openweights
 
 RUN apt-get update && \
-    apt-get install -y --no-install-recommends ca-certificates git openssh-server python3-venv wget \
-        psmisc procps lsof && \
-    wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb -O /tmp/cuda-keyring.deb && \
-    dpkg -i /tmp/cuda-keyring.deb && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends cuda-nvcc-12-8 && \
-    rm -f /tmp/cuda-keyring.deb && \
+    apt-get install -y --no-install-recommends ca-certificates git openssh-server python3-venv psmisc procps lsof && \
     rm -rf /var/lib/apt/lists/*
 
 # unison, for `ow ssh --sync`, which needs it present at both ends. Pinned to the upstream
@@ -46,18 +40,17 @@ RUN --mount=type=cache,target=/root/.cache/pip \
 
 RUN --mount=type=cache,target=/root/.cache/pip \
     /opt/venv/bin/python -m pip install --resume-retries 100 --timeout 180 \
-        "vllm==0.19.1"
+        "transformers==5.16.1"
 
 RUN --mount=type=cache,target=/root/.cache/pip \
-    /opt/venv/bin/python -m pip install --no-cache-dir --no-deps -e . && \
     /opt/venv/bin/python -m pip install --resume-retries 100 --timeout 180 \
         accelerate \
         bitsandbytes \
         datasets \
         PyJWT \
         cachier \
         diskcache \
-        fastapi \
+        "fastapi[standard]>=0.133.0,<0.137.0" \
         fire \
         hf_transfer \
         "httpx[http2]>=0.24.0" \
@@ -68,7 +61,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \
         runpod \
         scp \
         "supabase==2.15.3" \
-        trl
+        "trl==0.24.0" "vllm==0.28.0" -e . && \
+    /opt/venv/bin/python -m pip check
 
 RUN /opt/venv/bin/python - <<'PY'
 import importlib.metadata as metadata

README.md

@@ -68,7 +68,7 @@ class MyCustomJob(Jobs):
     }
     params: Type[BaseModel] = MyParams  # Your Pydantic model for params
     requires_vram_gb: int = 24
-    base_image: str = 'nielsrolf/ow-unsloth:v0.11' # optional
+    base_image: str = 'nielsrolf/ow-unsloth:v0.12.1' # optional
 
     def get_entrypoint(self, validated_params: BaseModel) -> str:
         # Get the entrypoint command for the job.

cookbook/custom_job/README.md

@@ -14,7 +14,7 @@ class MyCustomJob(Jobs):
     }
     params: Type[BaseModel] = MyParams  # Your Pydantic model for params
     requires_vram_gb: int = 24
-    base_image: str = 'nielsrolf/ow-unsloth:v0.11' # optional
+    base_image: str = 'nielsrolf/ow-unsloth:v0.12.1' # optional
 
     def get_entrypoint(self, validated_params: BaseModel) -> str:
         # Get the entrypoint command for the job.

experiments/training_comparison/LARGE_MODELS.md

@@ -0,0 +1,21 @@
+# Large-model feasibility (2026-09-05)
+
+## GLM-5.3
+
+The [official vLLM recipe](https://recipes.vllm.ai/zai-org/GLM-5.3) identifies GLM-5.3 as a roughly 743B-total/39B-active MoE. The default repository contains FP8 weights; BF16 has a separate model ID. The recipe requires vLLM 0.28.0+ and recommends eight 141GB H200/H20 GPUs for FP8 inference. It requires Transformers >=5.15.0. Its documented BF16 serving path needs multiple nodes.
+
+OpenWeights batch inference already uses all visible GPUs for tensor parallelism, except bitsandbytes loading. Consequently, a single 8xH200 worker is architecturally plausible for FP8 inference after updating the serving image. This has **not been GPU-validated here**. The provisioner also defaults to a 500GB volume, while batch inference downloads weights under `/workspace/hf_models`; a ~743GB FP8 checkpoint will not fit. Provision storage larger than the checkpoint plus download/runtime headroom before attempting this. The current generic model-size heuristic is unsuitable for giant MoEs; use an explicit VRAM requirement and hardware whitelist. Short-context smoke tests should precede throughput or long-context tests.
+
+Weight-only memory lower bounds (decimal GB), inferred from 743B parameters, are approximately 743GB FP8, 1,486GB BF16 and 371.5GB at ideal 4-bit packing. Actual memory is higher due to scales, unquantized tensors, KV cache, activation buffers and runtime overhead. The 39B active count does not describe weight storage.
+
+## Training
+
+The built-in OW loader forces a model onto one GPU (`device_map=None`, then `.to('cuda')`) and does not configure FSDP, ZeRO-3, tensor or expert sharding. `accelerate launch` and selecting multiple GPUs do not turn this into model-sharded training. DDP replicates model weights, so DDP alone cannot solve capacity.
+
+A viable future OW backend needs supported model sharding (FSDP/ZeRO-3 or a model-specific tensor/expert-parallel trainer), distributed launch and rank-aware logging/export. Full-parameter Adam training requires orders more memory than LoRA; even BF16 frozen-base LoRA needs all base weights resident or sharded. Quantized inference support is not proof of quantized training support.
+
+Tinker's live capabilities query returned `zai-org/GLM-5.3:peft:262144`. This is evidence of provider availability, not a tested training run or proof of parity with a future OW backend. No GLM GPU job has been launched during this pilot, preserving the preferred $100 budget.
+
+## Qwen3.8-27B
+
+The [Qwen model card](https://huggingface.co/Qwen/Qwen3.8-27B) describes a 27B dense multimodal model on the Qwen3.5 architecture. The [vLLM recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-27B) requests Transformers >=5.8.0. A one-H200 short-context BF16 LoRA smoke test is a reasonable capacity check; success on text data does not validate vision training. The Unsloth image resolves a different Transformers range than vLLM, so each image must be tested independently.

experiments/training_comparison/README.md

@@ -0,0 +1,66 @@
+# OpenWeights vs Tinker training diagnostics
+
+The goal is to detect broken optimization, masking, or checkpoint behavior that could compromise downstream research. Read [REPORT.md](REPORT.md) for observed results and limitations. Missing results are never filled with estimates.
+
+## Reproduce
+
+Use the repository environment with `tinker` installed. Supply `TINKER_API_KEY`; OW uses the project's `.env` and its organization credentials. All model uploads are private by default.
+
+```sh
+python experiments/training_comparison/common.py
+python experiments/training_comparison/run_tinker.py
+python experiments/training_comparison/run_openweights.py
+python experiments/training_comparison/run_openweights.py --collect
+python experiments/training_comparison/run_evaluation.py Qwen/Qwen3-8B
+# Replace MODEL with the trained adapter ID recorded in the OW manifest.
+python experiments/training_comparison/run_evaluation.py MODEL
+# Use the job ID printed above and the directory name of the corresponding run.
+python experiments/training_comparison/collect_evaluation.py JOB_ID ow-Qwen3-8B-sft-seed17
+python experiments/training_comparison/run_evaluation.py Qwen/Qwen3-8B --audit-only
+python experiments/training_comparison/report.py
+MPLCONFIGDIR=/tmp/ow-mpl python experiments/training_comparison/plot_results.py
+```
+
+The default is a small pilot: Qwen3-8B, rank 16, 128 training examples, batch 8, 32 updates, constant learning rate 1e-4, no warmup/weight decay/quantization. OW uses ordinary AdamW rather than its default 8-bit optimizer; RSLoRA is disabled. Tinker disables unembedding LoRA. Both train attention and MLP adapters. Tinker explicitly divides token weights by the number of supervised tokens in each batch because its cross-entropy loss is a sum.
+
+The explicit ChatML template removes automatic template differences from this first diagnostic. It does change the base model's expected thinking behavior. A follow-up with native templates is necessary before applying conclusions to ordinary production jobs.
+
+The seed-17 learning-rate sweep uses `--learning-rate 1e-5 --run-tag=-lr1e-5` and `--learning-rate 1e-3 --run-tag=-lr1e-3` with each training runner. Collect and evaluate the resulting OW adapter IDs as above. `learning-rate-sweep.png` plots only completed final evaluations, with alpha controls drawn as separate markers; the two-seed curves stay in `comparison.png`.
+
+## Artifacts
+
+- `data/`: deterministic train/test/shifted examples and SHA-256 hashes.
+- `results/tinker-*/`: configuration, token/mask audits, training metrics, evaluations at steps 0/16/32, and a resumable checkpoint path.
+- `results/ow-*/`: durable job ID, complete submitted configuration, status, events, and worker logs.
+- `worker_evaluation.py`: the same target-token mask and scoring for OW base models and HF adapters, plus greedy samples. Outputs job-relative `uploads/evaluation.json` through the normal OW artifact path.
+- `results/label-audit.json`: actual native SFT trainer/collator inputs and labels for all 128 training examples; all matched the comparison's shared encoding. The audit performs no optimizer updates.
+- `REPORT.md`: task-level accuracy, Wilson intervals and target-token NLL; selected successes and failures.
+
+Baseline and checkpoint evaluations use a 64-token generation cap. Record truncation and raw samples; formatting compliance and underlying reasoning capability are different measurements. The arithmetic task is intentionally reported separately from easy JSON extraction.
+
+## Completed audits
+
+- Two seeds at 1e-4 and a seed-17 sweep over 1e-5/1e-4/1e-3 on both backends (see REPORT.md).
+- Native SFT trainer/collator label audit: all 128 inputs/labels match the shared encoding (`results/label-audit.json`).
+- Adapter metadata audit: the Tinker export uses alpha 32 with rank 16 (`results/tinker-adapter-config.json`); OW defaults to alpha 16. The OW alpha-32 control at 1e-5 (`--lora-alpha 32 --run-tag=-lr1e-5-alpha32`) roughly halves the log-NLL gap to Tinker but leaves accuracy at 0%, so alpha is a partial explanation only.
+
+## Required follow-up before declaring equivalence
+
+1. Run at least three seeds per learning rate on both backends; the low-LR gap is currently a single seed.
+2. Extend the passing single-turn label audit to native templates, multiple turns, packing and gradient accumulation; compare identical minibatches.
+3. Check one-step loss/gradient/update parity using a small model with matched batch order. Verify LoRA initialization and scaling, target modules, optimizer epsilon and weight decay conventions; Tinker exposes neither alpha nor optimizer epsilon, so this needs a numeric probe rather than a config comparison.
+4. Add a small memorization task that should overfit, a shuffled-label negative control, native-template SFT, and preference-margin DPO evaluation.
+5. Compare exported checkpoint outputs through the same inference engine. Check base-model retention on tasks not in training.
+6. Report cost and time including startup/download, as well as steady-state training speed. Do not equate trainer-reported loss with a matched evaluation loss.
+
+## Qwen3.8 smoke tests
+
+```sh
+python experiments/training_comparison/run_openweights.py --model Qwen/Qwen3.8-27B --steps 2 --batch-size 1 --native-template --run-tag=-candidate-native --image nielsrolf/ow-unsloth:v0.12-candidate
+python experiments/training_comparison/run_openweights.py --model Qwen/Qwen3.8-27B --loss dpo --steps 2 --batch-size 1 --native-template --run-tag=-candidate-native --image nielsrolf/ow-unsloth:v0.12-candidate
+python experiments/training_comparison/run_inference_smoke.py
+```
+
+These test load/train/export, not training quality. Use `--image` to test a candidate image without changing the SDK's production defaults. Each OW job is constrained to one H200. Start sequentially when controlling peak GPU spending.
+
+Results for the v0.12 candidates (all passed after one DPO fix) are recorded in `results/image-validation.json`, `results/ow-Qwen3.8-27B-*-candidate-native*/` and `results/ow-Qwen3.8-27B-inference-candidate/`.