Skip to content

qwen36.md

Latest commit

 

History

History
88 lines (67 loc) · 3.73 KB

qwen36.md

File metadata and controls

88 lines (67 loc) · 3.73 KB
Raw

Qwen3.6-35B-A3B on colibri

c/qwen36.c runs Qwen/Qwen3.6-35B-A3B (35B total / ~3B active, Apache 2.0) — a hybrid architecture: 25% Gated Attention layers, 75% Gated DeltaNet (linear attention) layers, each followed by a streamed MoE block (256 experts, top-8 + 1 shared). Dense weights stay resident; routed experts stream from the container through an LRU + pinned cache. Development notes live in docs/qwen36-phase01.md / qwen36-phase02.md.

Architecture-identical checkpoints (same config geometry, e.g. KAT-Coder-V2.5-Dev) run on this engine unchanged.

Quickstart

Pre-converted containers (int4 experts, self-contained, ~20 GB):

# group-scaled int4 (gs64) — recommended, see "Which container" below
hf download Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 --local-dir ~/Models/qwen36_i4_gs64

# per-row int4
hf download Kreuzzelg/qwen36-35b-a3b-colibri-i4 --local-dir ~/Models/qwen36_i4

or convert the original bf16 checkpoint yourself (~70 GB download):

python3 c/tools/convert_qwen36.py --repo Qwen/Qwen3.6-35B-A3B --out ~/Models/qwen36_i4_gs64 --gs 64

Build and chat:

make -C c qwen36
COLI_MODEL=~/Models/qwen36_i4_gs64 ./c/coli chat

coli reads the model's config.json and matches its model_type against the family registry (qwen3_5_moe / qwen3_5_moe_text — an exact match, so other Qwen architectures are not claimed by this engine), picks it, and drives it over the serve protocol — coli web and coli serve (OpenAI-compatible API) work the same way.

Direct invocation without the gateway:

SNAP=~/Models/qwen36_i4_gs64 TOK=~/Models/qwen36_i4_gs64/tokenizer.json \
N_NEW=200 ./c/qwen36 256 4 prompt.txt

CPU prefill batching

On AVX2/FMA CPUs, dense int8 projections process two prompt rows per weight decode. The same kernel is used by attention, the router, DeltaNet projections, and the CPU shared expert. S=1 decode keeps the original four-register GEMV, and non-AVX2 builds retain the established per-row implementation. The shared expert additionally batches gate/up/down calls across prompt rows with at most 32 MiB of temporary activations; the CUDA expert tier keeps its per-token shared work so it can continue to overlap the in-flight GPU groups.

Both optimizations are bit-exact and on by default. For controlled A/Bs, QWEN_DENSE_BATCH=0 restores per-row dense-int8 GEMVs and QWEN_SHARED_BATCH=0 restores per-row shared-expert calls. A positive QWEN_SHARED_BATCH=N limits each shared chunk to N rows.

Requirements: ~30 GB RAM for comfortable expert caching and NVMe storage for the container. The default build is CPU-only; make -C c qwen36 CUDA=1 adds the optional CUDA VRAM expert tier documented in qwen36-cuda-tier.md.

Which container?

The gs64 container carries one scale per 64-weight group instead of one per row. On GLM, per-row int4 was the root cause of think-mode loops and never-terminating generations (#455), and group scales fixed them in controlled A/Bs — with moe_intermediate_size=512, Qwen's rows are short, so per-row quantization error concentrates the same way. The gs64 container costs ~1.7 GB more on disk and a few percent on cold-start; warm decode speed is the same or slightly better.

--ram is not honoured by this engine

coli --ram sizes the RAM budget for engines that stream experts from disk on demand. qwen36 does not: the CUDA expert tier it is built for (#713) requires full RAM residency of the expert set, so the budget is decided by the container, not by a flag. The engine reads no RAM_GB, and passing --ram changes nothing. Said here rather than silently ignored, because a flag that appears to work and does not is worse than one that is documented as unsupported.