feat(pflash): Qwen3.5-0.8B hybrid drafter with block-15 LongAttnComp scorer

[Graffioh]

Draft on upstream, opened from the fork branch Graffioh:codex/pflash-qwen35-hybrid (fork PR Graffioh#28). It carries the LongAttnComp core commits (codex/pflash-longattncomp-core, 10 commits) it is stacked on plus the Qwen3.5 hybrid drafter; the branch merges cleanly into main (dry-run merge, no conflicts) and is not yet rebased on it.

Evaluation of 10 September 2026 (full prefill vs Qwen3.5 PFlash)

Qwen3.6-27B target, real-use ratio budget (16K → ~3K, 120K → ~16K), 256-token chunks for QA and 1,024 for code, explicit question as scorer query, 8K output cap.

case set	full prefill	no head (original scorer)	untrained block-15 head	LongAttnComp-trained head (512 QA examples)
20 trial-0076 cases: QA 12 / code 8	10 / 6	3 / 0 (5 code answered)	10 / 6	10 / 6
12 differential cases: QA 4 / code 8	4 / 7	–	4 / 5	4 / 4
16 wide InfiniteBench CodeDebug cases	16	–	12	13
median time to first token QA / code	34 s / 215 s	15 s / 71 s	13 s / 43 s	13 s / 43 s

QA never loses a case to compression. CodeDebug losses (full 29/32, heads 23/32) come from the gold function not being selected (4 cases) or from a distractor definition cut by a 1,024-token chunk boundary, after which the target picks the truncated function (7 of 8 wrong-with-evidence answers). Offline, definition-aligned segment boundaries cut those truncations from 9 cases to 1 at the same budget; that runtime change is the next step on this branch. The original all-layer scorer on the Qwen3.5 backbone drops the evidence on nearly every case; the block-15 head is what makes the backbone viable.

Summary

PFlash can now use Qwen3.5-0.8B as its drafter with the LongAttnComp attention-mass scorer, the same selection mechanism the Qwen3-0.6B block-13 head uses. Under strict selection (PFLASH_LONGATTNCOMP_MODE=budget_only etc.) a Qwen3.5 drafter runs only its first 15 blocks (12 GatedDeltaNet + 3 full attention), then scores every context token against the query window with block 15's own NoPE Q/K projections: softmax over the keys before the query, mean over heads and query tokens, chunk-mass fill to the token budget. Qwen3.5's 262K native context covers the complete 76-97K-token CodeDebug inputs that the Qwen3-0.6B scorer cannot score inside its 32K window, and it shares the Qwen3.6 target tokenizer.

Runtime counterpart of the Python retention screen (trial 0075: untrained Qwen3.5 hybrid head 36/36 QA support retained vs 35/36 for both Qwen3 heads, 7/8 CodeDebug gold functions vs blocked for Qwen3).

Stacked on codex/pflash-longattncomp-core (merge base c927a59). The first commit carries, verbatim, the uncommitted research patches from the pflash-longattncomp-quality-probe worktree that trials 0023-0074 executed against (Qwen3 head loader, strict selection plumbing); every arm below runs on that state plus this branch.

What changed

• server/src/qwen3/qwen3_drafter.cpp
  • qwen35_longattncomp_score_and_compress: 15-block prefix, block-15 NoPE Q/K scoring, keys projected in 8K chunks (any tensor with the sequence length in a HIP grid y/z dimension aborts above 65,535 tokens; this failed at 87K before), one softmax over [S, query, heads], CPU mean reduction, then the shared select_longattncomp_chunks with direct chunk mass. Routed when strict selection is active; PFLASH_QWEN35_LEGACY_SCORER=1 keeps the old all-layer running-max scorer.
  • PFLASH_LONGATTNCOMP_HEAD_GGUF for this architecture: schema qwen3_5_0_8b_nope_qk_mass_v1, feature tap post_block14_residual_before_block15, query-only [1024,2048] and [1024,512] F32 tensors, drafter sha256 pin; fails closed on any mismatch, like the Qwen3 loader.
  • Drafter load plan skips the lm_head (tied-embedding Qwen3.5-0.8B exports omit output.weight; the old Qwen3.5 drafter branch could not load the 0.8B GGUF at all).
  • Causal mask builder memsets visible rows; DFLASH27B_KV_TQ3 guard is one RAII helper shared by both Qwen3.5 scorers.
• server/src/qwen35/gguf_target_loader.cpp: missing output.weight is only fatal when the load plan needs the lm_head.
• server/src/qwen3/qwen3_drafter_model.h: longattncomp_mean_token_mass helper (unit-tested in test_pflash_selection.cpp).
• server/README.md: drafter documentation.

Not in this PR: block-sparse FlashPrefill for head dimension 256 (Qwen3.5 attention runs dense ggml_flash_attn_ext), a trained Qwen3.5 head (the exporter lives in the training package), and drafter forward-pass optimisation (per-ubatch graph rebuilds; see timings).

Explicit scorer query (added after the comparison)

pflash_query (top level or extra_body) tells the compressor which text to score the context against, replacing the last-128-token message-tail heuristic; it must occur inside the latest user content and is matched unanchored with leading-token drift tolerance. Trial 0077 measured why this matters with the same untrained head at real-use budgets: the message tail keeps complete support on 24/36 QA and 5/8 CodeDebug gold functions, the explicit question keeps 32/36 and 7/8, and the retrieval-instruction prefix makes no difference. Runtime check on the ten flipping cases: the query window shrank from 128 tokens to the 11-37-token question and CodeDebug explicit choices went from 1/3 to 2/3 correct. The benches in luce_box PR #107 now send the recorded question per case.

Follow-ups on this branch (10 September 2026)

• Explicit query with a merged trailing token (135779b): a question ending in whitespace tokenizes to a trailing token on its own that merges with the prompt's newline in context; the unanchored window search only trimmed the query's leading tokens, so the mapping failed and the request was refused. Up to two trailing query tokens may now be dropped (untrimmed match preferred), and the expected query ids come from the matched range. Unit test added.
• Adaptive session keep ratio on this path (78e2530): the per-session acceptance-rate controller (adaptive_keep_ratio.h) already applies to the Qwen3.5 drafter, since the resolved ratio sizes the strict selector's budget and the feedback runs after every speculative turn. A new session started from a fixed 10% and ignored --prefill-keep-ratio / --prefill-curve; it now keeps the configured ratio until its first feedback and adapts from there within the same 2.5-20% range. README documents the behaviour and the caveat that acceptance is a proxy, not a retention measurement.
• 502 unit tests pass (test_server_unit, HIP gfx1201 build).

Verification

• 501 unit tests pass (test_server_unit), including the reduction, query-field and unanchored-query tests.
• Daemon smoke with Qwen3.5-0.8B-BF16.gguf (converted from the pinned HF revision 2fc06364…; norms already stored as 1+w): 31K-token QA case forward 4.1 s, chunk masses vs the Python head Pearson 0.999, top-16 chunk overlap 14/16; 87K-token CodeDebug case forward 21.5 s, gold function chunk retained, top-16 overlap 14/16.
• Head loader: exporting the native block-15 projections as a head GGUF and loading it through PFLASH_LONGATTNCOMP_HEAD_GGUF reproduces the head-less selection exactly (chunk-score difference 1.4e-4, identical chunks).

Quality and time comparison (trial 0076)

Target Qwen3.6-27B-Q4_K_M, hip:0, q4_0 KV, --max-ctx 137408, temperature 0, seed 20260908, thinking off, QA max_tokens 256, CodeDebug 2,048. Cases: reviewed development QA from the 0074 suite and the 8 complete CodeDebug cases (76-91K tokens). Arms: full prefill; Qwen3.5-0.8B hybrid (this PR, untrained head); Qwen3-0.6B PFlash with its native tail scorer under the same strict selection (the current runtime drafter). The Qwen3 trained-head arm was skipped to save time. Official benchmark scoring plus per-answer error-factor flags (luce_box PR #107). W&B run with per-answer tables: https://wandb.ai/logberto-na/lucebox/runs/ay4u28ti

Ratio policy (--prefill-curve 16384:0.1875 120000:0.1365, i.e. 16K→3K and 120K→16K; chunk 256 for QA, 1,024 for code). Accuracy is the official benchmark metric; "flagged" counts answers produced under an external error factor (request error or output cap), whose scores still count:

cohort	arm	official accuracy	flagged	median TTFT	median target prefill	median drafter	effective prompt tokens
QA (12)	Full prefill	10/12	0	34.1 s	33.9 s	n/a	23,024
QA (12)	Qwen3.5 hybrid	10/12	0	13.3 s	5.2 s	2.4 s	4,112
QA (12)	Qwen3-0.6B native	5/12	2 request errors	28.1 s	5.1 s	10-23 s	4,106
CodeDebug (8)	Full prefill	4/8	2 output-capped	235.1 s	234.7 s	n/a	92,813
CodeDebug (8)	Qwen3.5 hybrid	3/8	4 output-capped	46.3 s	17.4 s	23.0 s	13,376
CodeDebug (8)	Qwen3-0.6B native	3/8	7 output-capped	197.7 s	17.1 s	137-200 s	13,243

16K-budget policy (campaign protocol, constant 16,384-token budget, chunk 1,024; 30 reviewed QA cases): full prefill 27/30, Qwen3.5 hybrid 27/30 with the same per-case verdicts and no flags, median TTFT 44.0 s vs 29.6 s (1.1x median speedup, 0.78x on 17K prompts and 1.5x on 29K, because this budget barely compresses these prompts while a fixed 8-9 s compression cost is paid). Qwen3 arms were not run at this policy.

What the numbers say:

• QA quality. The untrained Qwen3.5 head reproduces the full-prefill official verdict on every QA case it ran on (42 cases across both policies), at 5x compression under the ratio policy with 2.6x lower TTFT. The Qwen3 native scorer loses half the answers at that budget.
• Qwen3 tokenizer mismatch. Two Qwen3 QA requests failed with final prompt exceeds target-token ceiling (5272 > 5253): the Qwen3 drafter tokenizer differs from the Qwen3.6 target tokenizer, so re-tokenized compressed text can overshoot by a few tokens and the server fails closed. Qwen3.5 shares the target tokenizer and cannot hit this.
• CodeDebug, stated explicitly. The runtime Qwen3-0.6B drafter does accept the 76-97K prompts, but that is beyond its documented 32,768-token context (40,960 configured positions); the offline Qwen3 scorer refuses them and trial 0069 found 0/8 gold functions retained there. Its rows are what the current runtime does, not a validated capability. The flagged CodeDebug answers all hit the 2,048-token output cap before an explicit choice (full 2/8, Qwen3.5 4/8, Qwen3 7/8), so code answer quality is confounded by the cap; on unflagged answers only, full prefill is 2/6 and Qwen3.5 1/4. Trial 0075 showed the hybrid head keeps the gold function on 7/8 of these inputs, so the gap to full prefill is mostly output truncation on compressed context. A rerun with a larger cap is needed before a code-quality conclusion.
• Time. In the native runtime the Qwen3.5 drafter is the faster one: 1.7-3 s forward on 17-29K prompts and 20-23 s on 76-97K, versus 10-23 s and 137-200 s for Qwen3-0.6B, whose time is almost entirely the sparse FlashPrefill HIP dispatch over 28 layers plus a 28-layer tail score. This reverses the Python-side speed gate (trial 0038), where Qwen3.5 lacked a GatedDeltaNet kernel. On CodeDebug Qwen3.5 PFlash is 5.1x faster than full prefill end to end (46 s vs 235 s).
• Remaining cost. Qwen3.5 CodeDebug TTFT is 23 s drafter forward + 17 s target prefill + about 6 s fixed overhead (drafter-tokenizer round trip, query mapping, re-tokenization). The forward rebuilds a graph per (layer, 1,024-token ubatch); that and the fixed overhead are the next targets. Block-sparse attention for head dimension 256 is not recommended before a profile shows attention dominating.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2xwWvXuXfzi7ZionCHKdk