model : fix k-pool indexer softmax gridDim.y overflow - #28730
Conversation
…ed up long context decode, fla, and slight MTP improvements.
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Reshape the k-pool gate logits before softmax so n_new does not map to gridDim.y, which is capped at 65535 on CUDA. Fixes prompt processing abort at n_kv >= 262144 reported in ggml-org#27773. Assisted-by: Cursor
Assisted-by: Cursor
|
Hi @AIalliAI, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
As I've replied on the other comments you made. Your instructions are conflicting and I will be combining all of these int one pr to support GLM5. @maintainers |
Overview
Reshape k-pool gate logits before
ggml_soft_maxson_newdoes not map togridDim.y, which is capped at 65535 on CUDA. During k-pool indexer prefill atn_kv >= 262144, the gate softmax is shaped[kpool, head_dim, n_new]and trips the launch atkpool = 4.Reshape to
[kpool, head_dim * n_new]before softmax, reshape back after. Same pattern as the comment inqwen4exp.cpp.Depends on #27773. Review the scoped 2-commit diff here:
timkhronos:GLM5.3-Flash...AIalliAI:fix/glm5-kpool-softmax-grid-y
Intended merge path: timkhronos#11 into
GLM5.3-Flash, then #27773 to master. Do not merge this PR directly to master.Additional information
Related: #27754, unslothai#214, #27901, #28175
Fork copies: timkhronos#11, AIalliAI#2
Requirements