Repository navigation
[AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: ROCm 10 image 20261102 + 10 upstream-PR patches, conc 1/2/12/24/48/80 / [AMD][MI35X] Qwen3.8-Flash-Next：ROCm 10 镜像 20261102 + 10 个上游 PR 补丁，并发 1/2/12/24/48/80 - #4832
ChangLiu0709 wants to merge 60 commits into
Conversation
…(upstream image) Add qwen3.8next-fp4-mi355x-sglang-agentic-mtp in the srt-slurm recipe layout: amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 on the upstream lmsysorg/sglang:v0.5.21-rocm724-mi35x image (pinned by digest), TP1/EP1, conc 1-16, native NEXTN MTP (3 steps, eagle-topk 1, 4 draft tokens) with the aiter attention backend. No local SGLang/AITER patches. Co-authored-by: Cursor <cursoragent@cursor.com>
…age at conc 8/12/16 Co-authored-by: Cursor <cursoragent@cursor.com>
…xt MI355X AgentX recipe Co-authored-by: Cursor <cursoragent@cursor.com>
…X AgentX recipe Co-authored-by: Cursor <cursoragent@cursor.com>
…10 image with patches Image: lmsysorg/sglang-rocm:v0.5.21-rocm10-mi35x-20261014 (pinned by digest). The recipe's setup_script patches its editable AITER (e7d2453f) and SGLang (bab9c81f) checkouts before the server starts: - ROCm/aiter#5992: FlyDSL causal conv1d prefill and GDN decode/verify - sgl-project/sglang#41879, ported to bab9c81f: opt-in SGLANG_AITER_GDN_CONV/_DECODE hooks - fused Triton QK Gemma-RMSNorm + M-RoPE + gate on HIP for Qwen3.5/Qwen3.8 A patch that no longer applies fails the worker. Local check on MI355X (TP1, recipe server args at conc 16, CI's lm-eval GSM8K, 5-shot, full test set): 0.9636 with GDN conv on and GDN decode off, as in the recipe; 0.9697 with GDN decode on too. Co-authored-by: Cursor <cursoragent@cursor.com>
…m 10 image amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 stores the MTP experts in BF16 but its Quark exclude list only names the MTP shared expert, so the draft model loaded the routed experts as MXFP4 and failed with a 1280 vs 2560 size mismatch. Apply the loader fix from amd-nprotaso/sglang qwen_3_8_improvement, which disables Quark for the draft when the checkpoint's MTP weights are unquantized. GSM8K (5-shot, recipe args, 1 GPU) with this checkpoint on the upstream image plus the setup script: 0.9697. Co-authored-by: Cursor <cursoragent@cursor.com>
… on MI355X Switch qwen3.8next-fp4-mi355x-sglang-agentic-mtp to amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8: MoE experts (including the shared and MTP experts) in OCP MXFP4, self-attention and linear-attention projections (including MTP self-attention) in per-channel FP8 with dynamic per-token FP8 activations, and the PLE table in per-tensor FP8. The upstream image's Quark W8A8 FP8 scheme serves the attention layers, and the draft keeps Quark since its MTP weights carry quantization scales. GSM8K (5-shot, recipe args, 1 GPU) on the upstream image plus the setup script: 0.9704; MTP mean accept length unchanged (3.49 vs 3.50). Co-authored-by: Cursor <cursoragent@cursor.com>
Add SGLANG_MAMBA_SSM_DTYPE=bfloat16 and SGLANG_USE_AITER_FP8_PER_TOKEN=1 to the server env, matching the checkpoint's reference setup. On this image the SSM dtype already follows --mamba-ssm-dtype, and the Quark W8A8 FP8 scheme takes per-token activation scaling from the checkpoint config. Co-authored-by: Cursor <cursoragent@cursor.com>
…ratio-3 packing into the MI355X image
Extend the recipe's setup script with five optimizations from
amd-nprotaso/{aiter,sglang} qwen_3_8_improvement and sgl-project/sglang#44302,
each as its own patch (AITER kernel, then the SGLang dispatch):
- FlyDSL hc_mix for the hyper-connection mix (SGLANG_AITER_HC_MIX, default
on) and the single-kernel hc_combine on ROCm (SGLANG_HC_COMBINE_SPLIT off)
- FlyDSL BF16 paged decode for QSA post-gather attention
(SGLANG_AITER_QSA_PA_DECODE, opt-in)
- single-kernel FlyDSL GDN chunked prefill (SGLANG_AITER_GDN_PREFILL=1 with
SGLANG_AITER_GDN_PREFILL_H=fused, opt-in)
- AITER topk_gating for the 512-expert softmax router
(SGLANG_ROCM_USE_AITER_TOPK_GATING, default on)
- fused QKVZ/BA packing for head ratio 3 (sglang#44302)
Later patches touch files of earlier ones, so the script now records the
applied patch set in a marker instead of reverse-checking each patch, and
resolves the patch directory to an absolute path. Applied to clean e7d2453f
and bab9c81f checkouts the result matches the source branches; a re-run is
a no-op. Not yet served end to end.
Co-authored-by: Cursor <cursoragent@cursor.com>
Serve with --ple-offload-embedding and declare the fp8 PLE storage through --json-model-override-args (text_config.ple_embedding_dtype=float8_e4m3fn): SGLang rejects switching the offloaded table to fp8 at load. Frees the table's GPU memory for KV cache. Not yet verified on MI355X. Co-authored-by: Cursor <cursoragent@cursor.com>
…loading and GDN decode
Three more patches from amd-nprotaso/{aiter,sglang} qwen_3_8_improvement for the
upstream ROCm 10 image:
- sglang 7e3eb83f97: padded QSA group writes read the inert source slot, off-page
prefixes use the absolute mamba checkpoint grid, separate shared-expert tensors
load into the fused MoE slot (strict for the MTP draft)
- sglang d309df65df: unfinished prefill chunks end on a QSA page under exact
chunk fill (on by default for gfx95); tests ported to this image's Req API
- aiter badcd62a: faster FlyDSL GDN decode/verify (SGLANG_AITER_GDN_DECODE)
Verified locally on the upstream image: all 15 patches apply, unit tests from
the source commits pass (77), FlyDSL GDN decode matches Triton for decode and
MTP verify, GSM8K 0.9742 (GDN decode off) / 0.9689 (on) with PLE offload, and
two-turn long-prompt retrieval up to ~90k tokens is 20/20.
Co-authored-by: Cursor <cursoragent@cursor.com>
…nFP8 checkpoint, ROCm 10 image patches and PLE offload Co-authored-by: Cursor <cursoragent@cursor.com>
…ant env - sglang-unquantized-mtp-draft.patch only disabled Quark for a fully floating-point MTP branch; the AttnFP8 checkpoint's MTP experts are MXFP4 with scales, so it never fired. The next SGLang patch is re-exported for the shifted qwen3_5_mtp.py offsets; its content is unchanged. - SGLANG_MAMBA_SSM_DTYPE is set from --mamba-ssm-dtype at startup. - SGLANG_USE_AITER_FP8_PER_TOKEN is read only by Fp8LinearMethod; the checkpoint's per-channel FP8 attention is served by QuarkLinearMethod. Local GSM8K on the CI recipe with PLE offload, 3 runs each: 0.9709 before, 0.9697 after (same-setup spread 0.9659-0.9742); MTP accept length unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
…ode recipe The patches move to qwen3.8next/sglang/mi355x-fp4-mtp/patches/ next to agentic.yaml. srtctl looks up setup_script only in configs/, so the entry script stays there and reads the patches through the /infmax-workspace mount. Co-authored-by: Cursor <cursoragent@cursor.com>
… and fused GDN prefill GSM8K stays within same-setup noise with all three on (0.9735, 0.9659 vs 0.9697), accept length 3.49, and long-context two-turn retrieval 20/20. Co-authored-by: Cursor <cursoragent@cursor.com>
…provement Add the fused FlyDSL QSA sparse decode, the fused GDN decode proj+conv on HIP with off-by-default GDN prefill opt-ins, QSA paged decode on AITER mainline's planned pa_decode, and FlyDSL hc_mix up to 128 rows (sglang 0c3cedfc32, 1275650163, 67750e35dc; aiter 47d91306, a4909abc). The mainline pa_decode keeps this image's FlyDSL 0.3.4.1 buffer_ops and kernels_common. Co-authored-by: Cursor <cursoragent@cursor.com>
…uned GEMM tables sglang 6c6300f49f as an off-by-default patch (shared attention input quant, fused GDN norm + out_proj quant, shared-expert gate in the router, lm_head through tgemm) and aiter 38e2650b's Qwen3.8 gfx950 tuned GEMM tables, which AITER loads by default. aiter ee5f7a82 (fused MoE sort + quant) changes C++ sources and is left out: the setup script does not rebuild AITER. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…spec-verify Turn on SGLANG_OPT_QWEN4_SHARED_ATTN_QUANT, _QWEN4_FUSED_GDN_NORM_QUANT, _QWEN_MOE_FUSED_SHARED_TOPK and _AITER_LM_HEAD_GEMM, AITER_GDR_FLYDSL, --enable-linear-replayssm-spec and --tokenizer-worker-num 6. The fused GDN norm quant needs the sigmoid gate from aiter #5392, added as a patch whose JIT module is rebuilt in the container. Co-authored-by: Cursor <cursoragent@cursor.com>
…branches Move to lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261018, whose AITER (2bb54910) already carries the FlyDSL QSA pa_decode and sigmoid-gated norm quant patches, so those two are dropped. Add Christian's fused decode MoE route-sort + MXFP4 quant (aiter #6165, #6467, ee5f7a82) with the mainline fused_moe.py it depends on (aiter #6225, #5804, #6050), and Nikolai's MTP draft quantization resolution from checkpoint headers. Co-authored-by: Cursor <cursoragent@cursor.com>
Add the Qwen4-Exp FlyDSL monokernel from amd-nprotaso/sglang qwen_3_8_improvement (7aca04ba83, 657dcf9ee5) as an SGLang patch and turn it on for target verify and the MTP draft. Its verify path writes the per-draft intermediate SSM states, which ReplaySSM spec-verify does not allocate, so --enable-linear-replayssm-spec is dropped. Co-authored-by: Cursor <cursoragent@cursor.com>
…-static 0.9 Co-authored-by: Cursor <cursoragent@cursor.com>
… at c8+ Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… at c16 Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ction-static 0.85 Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…upstream PRs; c12 on HiCache, fused GDN prefill AITER 01-03 (ROCm/aiter#5992, #6670, #6496) and SGLang 04-10 (sglang#48648, #45196, #48649, #45529, #41683, #41879, #44103), source files only, applied in order by qwen3.8next-rocm10-patches.sh on the 20261102 image. Patch 10 forwards the image's **kwargs to the new _extend() and skips the AITER prefill for a captured prefill graph. SGLANG_AITER_GDN_PREFILL_H=fused selects the single-kernel prefill of aiter#6496. c12 joins c24/c48/c80 on FP8 KV + HiCache; c1-c8 and c16 keep GPU-resident KV.
GPU-resident KV at c1 and c2 (monokernel); FP8 KV + HiCache at c12, c24, c48 and c80. Drop the c4, c8 and c16 points and their recipe overrides.
…refactor # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @ChangLiu0709's task in 1m 2s —— View job Review of PR #4832
1 blocking issue. In The rest of the diff looks correct:
|
ChangLiu0709
added
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| scheduler-recv-interval: 10 | ||
| stream-interval: 50 | ||
| reasoning-parser: qwen3 | ||
| default-chat-template-kwargs: '{"enable_thinking": false}' |
There was a problem hiding this comment.
🔴 BLOCKING: This line adds back --default-chat-template-kwargs '{"enable_thinking": false}', and because it sits in base, it applies at every concurrency. #4542 removed this flag on purpose (perf-changelog.yaml:11886): "Every point now serves the model's default thinking-on chat template … so the served mode matches the thinking_on golden AL (2.32 at 3 draft tokens) and the other qwen3.8next recipes."
Why it matters: The AgentX replay would run with thinking off on this recipe only. That changes output lengths and MTP acceptance, so these results can't be compared with #4779, earlier runs of this config key, or the other qwen3.8next recipes. The golden AL file qwen3.8-flash-next-fp8__mtp__native__default-default.yaml has only a thinking_on entry. Neither the PR description nor the new changelog entry mentions this change, so it looks unintended. It may have been carried over from the AMD submission recipes.
Fix: Remove the line. If thinking-off is intended, say so in the changelog entry and explain why this recipe should differ from the others.
| default-chat-template-kwargs: '{"enable_thinking": false}' |
| scheduler-recv-interval: 10 | ||
| stream-interval: 50 | ||
| reasoning-parser: qwen3 | ||
| default-chat-template-kwargs: '{"enable_thinking": false}' |
There was a problem hiding this comment.
🔴 Every point of this AgentX recipe, and its GSM8K evals, now runs Qwen3.8 with reasoning turned off. Neither the PR description nor the changelog entry says so. The new server default at line 51 applies to every request because the recipe sets no AIPERF_EXTRA_INPUTS chat_template_kwargs. Other agentic recipes enable thinking explicitly. Fix: drop this default, or state in the perf-changelog entry that thinking is off. Shorter outputs change throughput, so results would not be comparable with the base run or with other thinking-on recipes.
Why this was flagged
agentic.yaml:51 adds default-chat-template-kwargs: '{"enable_thinking": false}' under base roles.agg.args, so all six points (c1, c2, c12, c24, c48, c80) inherit it. The benchmark env in the recipe (lines 78-80) sets only AIPERF_REQUIRED_SERVER_METRIC_PREFIX and AIPERF_APPLY_CHAT_TEMPLATE, with no per-request chat_template_kwargs. The server default therefore decides thinking mode for both the AgentX replay and the eval. On the base branch the recipe had no such default, so the Qwen template used thinking-on, its default. Other agentic recipes pass {"chat_template_kwargs":{"enable_thinking":true}} explicitly (glm5.2/sglang/b200-fp8-mtp/agentic.yaml:52, qwen3.5/sglang/gb200-fp4/agentx/agg-variants.yaml:84). The new perf-changelog.yaml block for PR #4832 never mentions that thinking is off. Ingested results would show the shorter non-reasoning output profile with no record of why.
Verification: Trigger: every run of qwen3.8next-fp4-mi355x-sglang-agentic-mtp after this merge. All six points and their GSM8K evals change from thinking-on to thinking-off, and neither the changelog nor the PR description says so. agentic.yaml:51 adds default-chat-template-kwargs: '{"enable_thinking": false}'. perf-changelog.yaml:11886 (PR #4542) removed this exact flag on purpose for this same config key.
| apply /sgl-workspace/aiter 01-aiter-5992-flydsl-causal-conv1d-prefill.patch | ||
| apply /sgl-workspace/aiter 02-aiter-6670-flydsl-qsa-sparse-decode.patch | ||
| apply /sgl-workspace/aiter 03-aiter-6496-flydsl-gdn-fused-prefill.patch | ||
| apply /sgl-workspace/sglang 04-sglang-48648-qwen4-fused-hc-decode.patch | ||
| apply /sgl-workspace/sglang 05-sglang-45196-qsa-flydsl-sparse-decode.patch | ||
| apply /sgl-workspace/sglang 06-sglang-48649-qsa-decode-mqa-launch.patch | ||
| apply /sgl-workspace/sglang 07-sglang-45529-qwen35-fuse-selfattn-prepare.patch | ||
| apply /sgl-workspace/sglang 08-sglang-41683-gdn-prefill-strided-views.patch | ||
| apply /sgl-workspace/sglang 09-sglang-41879-aiter-gdn-conv-prefill.patch | ||
| apply /sgl-workspace/sglang 10-sglang-44103-aiter-gdn-chunked-prefill.patch |
There was a problem hiding this comment.
🔴 CODEOWNER sign-off Check 10 will FAIL for this PR. It patches the pinned image's AITER and SGLang at runtime but adds no engine-patch waiver. The setup script runs git apply on 10 .patch files, which is exactly the kind of engine patching the rule bans without a waiver. No inferencex-e2e/docs/waiver/4832.md is in the diff. Fix: either add that waiver (what is patched, why the stock image cannot run this benchmark, the upstream PR links and a removal plan) and link it from the sign-off, or drop the patches and the setup_script.
Why this was flagged
agentic.yaml:22 wires setup_script: qwen3.8next-rocm10-patches.sh for every point of qwen3.8next-fp4-mi355x-sglang-agentic-mtp. Lines 47-56 of that script run git apply with 3 AITER and 7 SGLang patches against /sgl-workspace/aiter and /sgl-workspace/sglang before the server starts. .github/codeowner-signoff-verify-prompt.md:482-525 (Check 10) lists .patch files, git apply and setup_script wiring as patching that reaches the serving stack. Such patching FAILs unless a waiver exists at inferencex-e2e/docs/waiver/<PR_NUMBER>.md, is filed in this PR, and is linked from the sign-off. PR_REVIEW_CHECKLIST.md:71 repeats the same rule. The only waiver in docs/waiver/ is 2508.md, and this diff adds none. On the base branch this config ran the image unpatched: base agentic.yaml:3 said "The upstream image is used unpatched".
Verification: The base recipe sets setup_script: qwen3.8next-rocm10-patches.sh (agentic.yaml, about line 22), so c1/c2 and c12 through c80 all reach it. .github/codeowner-signoff-verify-prompt.md:482-525 (Check 10) says "The pinned upstream image must run AS SHIPPED". The diff touches 14 files and adds no waiver. inferencex-e2e/docs/waiver holds only 2508.md.
| # packing (sglang#44302), the 128-row hc_mix (sglang#41880) and ROCm/aiter#6225, #5804, | ||
| # #6050, #6165, #6467, so none of them is patched. All Python. A patch that no longer | ||
| # applies fails the worker. | ||
| set -euo pipefail |
There was a problem hiding this comment.
🟡 (optional) The new setup script turns on nounset, which the repo's mandatory Bash rules ban. Every other setup script in this directory uses set -eo pipefail. AGENTS.md:48 bans set -u, set -o nounset and set -euo pipefail in active scripts. Fix: change line 24 to set -eo pipefail. Nothing in the script relies on unset-variable errors, since patches, marker and digest are all assigned before use.
Why this was flagged
AGENTS.md:48 says: "No nounset: no set -u, set -o nounset, set -euo pipefail, or bash -u. Keep other options such as set -eo pipefail." The new script qwen3.8next-rocm10-patches.sh:24 uses set -euo pipefail. The sibling setup scripts in the same directory (glm5.3-tilert-rocm.sh:2 and the others) use set -eo pipefail. The base branch has no such script, so this PR adds the violation. Nounset is banned because it breaks unset-safe probes and check_env_vars handling. Dropping -u here changes nothing else, because every variable the script reads is assigned locally.
Verification: nit. Line 24 of inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/qwen3.8next-rocm10-patches.sh is set -euo pipefail. AGENTS.md:48 bans set -euo pipefail in active scripts. The base has no such file, so the violation is new. Every variable in the script is assigned before it is used, so -u never fires.
| # Qwen3.8-Flash-Next kernels on lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261102, | ||
| # applied to its editable AITER (285579b6) and SGLang (1ced9a9eb0) checkouts. Each | ||
| # patch is the source-only diff of an open upstream PR (tests, benchmarks and AITER's | ||
| # AOT registry dropped; the header of each file names the PR and its head commit). | ||
| # AITER | ||
| # - 01 ROCm/aiter#5992 FlyDSL causal conv1d prefill, GDN decode/verify kernels | ||
| # - 02 ROCm/aiter#6670 FlyDSL fused QSA sparse decode (needed by sglang#45196) | ||
| # - 03 ROCm/aiter#6496 FlyDSL single-kernel GDN fused chunked prefill | ||
| # SGLang | ||
| # - 04 sglang#48648 fused decode-sized hyper-connection norm + inject, SGLANG_OPT_QWEN4_FUSED_HC | ||
| # - 05 sglang#45196 QSA sparse decode on the AITER FlyDSL kernel (on with SGLANG_USE_AITER) | ||
| # - 06 sglang#48649 faster QSA decode MQA logits (padded Q, group-strided launch) | ||
| # - 07 sglang#45529 fused QK norm + RoPE + gate for Qwen3.5 self-attention on gfx950 | ||
| # - 08 sglang#41683 strided GDN prefill projection views on HIP | ||
| # - 09 sglang#41879 GDN prefill conv on AITER FlyDSL, SGLANG_AITER_GDN_CONV | ||
| # - 10 sglang#44103 GDN chunked prefill on AITER, SGLANG_AITER_GDN_PREFILL[_H] | ||
| # AITER 01 is needed by 09, 02 by 05 and 03 by 10 with SGLANG_AITER_GDN_PREFILL_H=fused, | ||
| # so the AITER patches go first. The image already has the monokernel | ||
| # (--enable-amd-monokernel), the AITER topk_gating router (sglang#40708), ratio-3 GDN | ||
| # packing (sglang#44302), the 128-row hc_mix (sglang#41880) and ROCm/aiter#6225, #5804, | ||
| # #6050, #6165, #6467, so none of them is patched. All Python. A patch that no longer | ||
| # applies fails the worker. |
There was a problem hiding this comment.
🟡 nit (optional): the new script, recipe and master-config comments include header and provenance text that AGENTS.md:56-57 bans; sweep:patched into the upstream ROCm 10 image|^# Qwen3.8-Flash-Next kernels on|# Concurrency 1 and 2 serve from GPU KV Examples: qwen3.8next-rocm10-patches.sh:2-23 is a 22-line file header listing PRs and what the image already ships. agentic.yaml:18-21 says where the kernels came from. Fix: remove them and keep only short line-level technical comments, such as the marker rationale at lines 29-30.
Why this was flagged
AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." AGENTS.md:57 says "No history: how code evolved, what it replaced, where it came from, or past results. Git and perf-changelog.yaml are the record." The script header at qwen3.8next-rocm10-patches.sh:2-23 describes the file, lists upstream PR provenance, and names what the image already contains. The recipe at agentic.yaml:18-21 gives PR provenance for setup_script. The new section comment at agentic.yaml:124-132 describes the c12-c80 overrides. The amd-master.yaml comment added before the qwen3.8next key describes the config's concurrency layout. All of these lines are new in this diff. The perf-changelog entry already records the same provenance.
Verification: This applies whenever the change merges as written. CLAUDE.md delegates all guidance to AGENTS.md. AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." AGENTS.md:57 says "No history: how code evolved, what it replaced, where it came from, or past results." qwen3.8next-rocm10-patches.sh:2-23 is a 22-line file header. Nothing breaks at runtime.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42719632820 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42719632820 |
…refill breakable at c12-c80 The AMD reference recipes moved to a 262144-token Mamba checkpoint interval and breakable prefill CUDA graphs (their patch 22 makes breakable the ROCm default; this image lacks it, so the flag is passed explicitly). The changelog entry says so.
…refactor # Conflicts: # inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/mi355x-fp4-mtp/agentic.yaml # inferencex-e2e/configs/amd-master.yaml # inferencex-e2e/perf-changelog.yaml
… c80 c12 and c24 move to the kv-offloading none search-space entry (conc-list 1, 2, 12, 24) and lose enable-hierarchical-cache, hicache-size, hicache-io-backend, hicache-mem-layout and hicache-write-policy; c48 and c80 keep them (hicache-size 298). FP8 KV, the 262144 track interval, breakable prefill graphs and fused HC stay at c12-c80.
…refactor # Conflicts: # inferencex-e2e/perf-changelog.yaml
…, drop the three AITER patches The setup script pip-installs the sha256-pinned v0.1.29 ROCm 7.14 / cp312 wheel (no deps) and flydsl 0.4.2 over the image's editable AITER, checks that python imports the wheel, then applies the seven SGLang patches (renumbered 01-07). v0.1.29 contains aiter#5992, #6670, #6496 and the hc_mix kernels. The ROCm 10.0 and 10.1 wheels fail on any tensor call with the image's torch (built for HIP 7.15); the 7.14 build works.
…upport) as patch 08 Co-authored-by: Cursor <cursoragent@cursor.com>
…refactor # Conflicts: # inferencex-e2e/perf-changelog.yaml
Summary
Qwen3.8-Flash-Next Quark MXFP4 (FP8 attention, FP8 PLE) AgentX recipe on MI355X, SGLang native NEXTN MTP, TP1/EP1, on the upstream ROCm 10 image
lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261102@sha256:acf1fee05096c6eb757d4642e3b87c36766bce1334277e49b917afc2f06c7b5a(SGLang1ced9a9eb0, AITER285579b6). A recipesetup_scriptinstalls the AITER v0.1.29 release wheel over the image's AITER and applies 8 SGLang patches for kernels that are still open upstream, and the recipe runs concurrency 1, 2, 12, 24, 48 and 80. This is the patched counterpart of the unpatched baseline in #4779.qwen3.8next-fp4-mi355x-sglang-agentic-mtp:kv-offloading: nonewithconc-list: [1, 2, 12, 24], andkv-offloading: dram(backendhicache) withconc-list: [48, 80], oneagentic.yamlrecipe. The matrix generator yields exactly these six points. c4, c8 and c16 are dropped.qwen3.8next-rocm10-patches.shrunspip install --no-depsonAITER_WHEEL_URL(the v0.1.29 ROCm 7.14 cp312 wheel, pinned by URL and sha256 in the recipeenvironment) and onflydsl==$AITER_FLYDSL_VERSION(0.4.2: the wheel requires it, the image has 0.4.0). It then checks thatpip checkreports nothing for AITER or FlyDSL and that Python imports the wheel (not the editable checkout in/sgl-workspace/aiter) with the FlyDSL conv, QSA, GDN fused prefill and hc_mix modules present. v0.1.29 contains ROCm/aiter#5992 (FlyDSL causal conv1d prefill, GDN decode/verify), #6670 (fused QSA sparse decode, needed by sglang#45196), #6496 (single-kernel GDN fused chunked prefill) and #5993 (hc_mix), which the previous revision of this PR patched into the image's AITER (Oct 17). The three AITER patches are gone.2.11.0+rocm10.0.0but built for HIP 7.15. The ROCm 10.0 and 10.1 wheels of the same release have prebuilt kernels that fail on any tensor call with it: the first smoke run with the 10.0 wheel segfaulted inaiter::torch_itfs::mha_varlen_fwd(vision-tower warm-up), and a repro in the image raised "FlashAttention only support fp16, bf16 and fp8_e4m3 data type" for plain bf16 tensors. The same call works after a JIT rebuild (AITER_REBUILD=1) and with the ROCm 7.14 wheel, which is pinned.benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/mi355x-fp4-mtp/patches/), each the source-only diff of an open PR (tests, benchmarks and docs dropped; the header of each file names the PR and its head commit). The setup script applies them in order withgit apply, is idempotent through a marker that also covers the wheel pin, and fails the worker if one stops applying.SGLANG_OPT_QWEN4_FUSED_HC)SGLANG_AITER_GDN_CONV)SGLANG_AITER_GDN_PREFILL,_H=fused)extend()intoextend()and_extend(), but the image'sextend()body also reads**kwargs(CUDA-graph chunk tables), sokwargswas undefined in_extend()and every GDN prefill would raiseNameError. Patch 07 forwards**kwargsand skips the AITER prefill for a captured prefill graph. The header of the file says so.mainwhose top-p rescale helper (_adjust_top_p_for_top_k, sglang#41916) the image lacks, so the helper is copied intoeagle_utils.py. The path turns on by default whenSGLANG_USE_AITERis set and the MTP verify uses rejection sampling (NEXTN with eagle-topk 1); every AgentX request carriestop_k20 from the model'sgeneration_config.json, and requests without a top-k of 2 to 128 keep the dense path. It needs no recipe flag.SGLANG_USE_AITER,SGLANG_AITER_GDN_CONV,SGLANG_AITER_GDN_PREFILLandSGLANG_AITER_GDN_PREFILL_H=fusedat every point (the sglang#44103 default is the triton stages;fusedselects the single kernel of aiter#6496).SGLANG_OPT_QWEN4_FUSED_HC,SGLANG_USE_AITER_UNIFIED_ATTNandSGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTIONat c12/c24/c48/c80. The FlyDSL QSA decode has no flag: it is on wheneverSGLANG_USE_AITERis set on gfx950.mem-fraction-static0.88 as given, anextra_buffer_lazyMamba pool (mamba-track-interval262144,max-mamba-cache-size960 / 960 / 720 / 640), and, at c48 and c80 only, HiCache (kernelIO,page_first,write_back,hicache-size298); c12 and c24 serve from GPU KV without HiCache. Admission is 24 at c12, 48 at c24 and 128 at c48 and c80. c12-c80 also pass--cuda-graph-backend-prefill breakable(the AMD reference recipes get it from an SGLang default change this image lacks), and c80 setsstrip-thinking-cacheandmamba-max-states-per-path 2. c1 and c2 are unchanged.inferencex-e2e/perf-changelog.yaml.Validation
AITER_WHEEL_URLis the v0.1.29 ROCm 7.14 cp312 wheel with a sha256 fragment andAITER_FLYDSL_VERSIONis the flydsl that release pins. The earlier revision's gate found the patch 07kwargsbug.flash_attn_varlen_funcon bf16 inputs fails with the ROCm 10.0 and 10.1 wheels and passes with the ROCm 7.14 wheel (head dims 64, 72 and 128).srt-slurm: all checks passed at c1 and c12 (7 patches) and at c12 and c48 (8 patches). Each run installedamd-aiter 0.1.29+rocm7.14andflydsl 0.4.2(imported from/opt/venv), applied 7/7 patches (8/8 in the later c12 and c48 runs), started the server in about 7 minutes with no runtime JIT builds, logged "Using AITER fused FlyDSL GDN chunked prefill", had no engine traceback and no FlyDSL fallback, and had the recipe env in the server; c12 had no HiCache flags, FP8 KV, track interval 262144 and breakable prefill graphs on the command and the server did not disable them.c1 has the same settings in both runs, so the wheel changes its throughput by -0.1% and p90 interactivity by -0.1%. The two c12 rows differ in more than AITER (HiCache, track interval, breakable prefill graphs), so they are not an A/B of the wheel; c24 was not rerun after HiCache was dropped there. Adding patch 08 changes c12 by +0.1% throughput and -0.7% p90 interactivity, which is within run-to-run noise: the saving the PR measures (about 2 ms of an eager decode step at 12 requests) is not visible in this replay, and the logs do not show whether the sparse path ran. c48 has no earlier run on a comparable setup. An earlier c1 run with admission 2 (2x CONC) measured 13,467 tok/s/GPU and p90 175.2, which is why c1 and c2 keep admission = CONC. For comparison, the unpatched baseline in #4779 measured 53,785 tok/s/GPU and p90 90.8 at c12 (GPU KV, no HiCache) and 107,298 and 50.4 at c24. These are single local runs, not CI numbers. Not yet run with this setup: c2, c24, c48, c80, GSM8K, and the e2e Test workflow.
AI model disclosure
PR checklist and change type
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
摘要
Qwen3.8-Flash-Next Quark MXFP4（FP8 attention、FP8 PLE）AgentX 在 MI355X 上的 recipe，SGLang 原生 NEXTN MTP，TP1/EP1，使用上游 ROCm 10 镜像
lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261102@sha256:acf1fee0…（SGLang1ced9a9eb0，AITER285579b6）。recipe 的setup_script在镜像的 AITER 之上安装 AITER v0.1.29 release wheel，并为仍在上游审核中的内核应用 8 个 SGLang 补丁，扩展到并发 1、2、12、24、48、80。本 PR 是 #4779（无补丁基线）的带补丁版本。qwen3.8next-fp4-mi355x-sglang-agentic-mtp：kv-offloading: none（conc-list: [1, 2, 12, 24]）与kv-offloading: dram（后端hicache，conc-list: [48, 80]），共用一个agentic.yaml。矩阵生成器恰好产生这六个点，c4、c8、c16 已移除。qwen3.8next-rocm10-patches.sh对AITER_WHEEL_URL（v0.1.29 的 ROCm 7.14 cp312 wheel，在 recipe 的environment中以 URL 与 sha256 固定）和flydsl==$AITER_FLYDSL_VERSION（0.4.2：wheel 要求该版本，镜像为 0.4.0）执行pip install --no-deps，并检查pip check对 AITER/FlyDSL 无报错、Python 导入的是 wheel（而非/sgl-workspace/aiter的 editable 目录）且 FlyDSL conv、QSA、GDN fused prefill、hc_mix 模块存在。v0.1.29 已包含 [FlyDSL] [gfx950] Causal conv1d prefill and GDN decode/verify kernels for Qwen3.5/Qwen3.8 ROCm/aiter#5992、#6670、#6496 与 #5993（hc_mix），此前版本的 PR 曾把它们作为补丁打进镜像的 AITER（10 月 17 日）；三个 AITER 补丁已移除。2.11.0+rocm10.0.0，但针对 HIP 7.15 构建。同一 release 的 ROCm 10.0 与 10.1 wheel 的预编译 kernel 在该 torch 下任何 tensor 调用都会失败：首次用 10.0 wheel 的 smoke 在aiter::torch_itfs::mha_varlen_fwd（视觉塔 warm-up）段错误，镜像内复现对普通 bf16 tensor 报 "FlashAttention only support fp16, bf16 and fp8_e4m3 data type"。JIT 重编译（AITER_REBUILD=1）与 ROCm 7.14 wheel 下同一调用正常，因此固定使用 7.14。git apply按序应用，通过同时覆盖 wheel 版本的标记文件保证幂等，补丁失效则 worker 失败。PR：sglang#48648、#45196、#48649、#45529、#41683、#41879、#44103、#48090（编号 01–08）。extend()拆为extend()与_extend()，而镜像的extend()函数体还会读取**kwargs（CUDA graph chunk 表），导致_extend()中kwargs未定义、每次 GDN prefill 都会抛NameError。补丁 07 透传**kwargs，并在 prefill graph 捕获时跳过 AITER prefill。main，其 top-p 重缩放函数（_adjust_top_p_for_top_k，sglang#41916）镜像中没有，因此将该函数复制进eagle_utils.py。设置SGLANG_USE_AITER且 MTP verify 使用 rejection sampling（NEXTN、eagle-topk 1）时默认启用；AgentX 的每个请求都带有模型generation_config.json中的top_k20，没有 2–128 之间 top-k 的请求仍走原稠密路径。无需 recipe 参数。SGLANG_USE_AITER、SGLANG_AITER_GDN_CONV、SGLANG_AITER_GDN_PREFILL与SGLANG_AITER_GDN_PREFILL_H=fused；c12/c24/c48/c80 另设SGLANG_OPT_QWEN4_FUSED_HC、SGLANG_USE_AITER_UNIFIED_ATTN、SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION。mem-fraction-static0.88、extra_buffer_lazyMamba 池（mamba-track-interval262144）、可中断（breakable）prefill CUDA graph（--cuda-graph-backend-prefill breakable）；仅 c48、c80 启用 HiCache（hicache-size为 298），c12、c24 不使用 HiCache，仅使用 GPU KV。c12/c24/c48/c80 的服务器设置来自 AMD Qwen3.8 提交 recipe，不含其额外补丁。inferencex-e2e/perf-changelog.yaml末尾追加一条记录。验证
AITER_WHEEL_URL是带 sha256 的 v0.1.29 ROCm 7.14 cp312 wheel，且AITER_FLYDSL_VERSION与该版本固定的 flydsl 一致。flash_attn_varlen_func在 ROCm 10.0、10.1 wheel 下失败，在 ROCm 7.14 wheel 下通过（head dim 64、72、128）。srt-slurm）：c1、c12（7 个补丁）以及 c12、c48（8 个补丁）的所有检查均通过。每次运行安装了amd-aiter 0.1.29+rocm7.14与flydsl 0.4.2（从/opt/venv导入），应用 7/7 个补丁（后续 c12、c48 运行为 8/8），服务器约 7 分钟启动且无运行时 JIT 编译，日志出现 "Using AITER fused FlyDSL GDN chunked prefill"，无引擎 traceback，无 FlyDSL 回退，服务器环境变量与 recipe 一致；c12 命令行无 HiCache 参数，包含 FP8 KV、track interval 262144 与 breakable prefill graph，且服务器未将其关闭。结果见上方英文表格。c1 在两次运行中设置相同，wheel 使吞吐变化 -0.1%、p90 interactivity 变化 -0.1%。两行 c12 的差异不止 AITER（HiCache、track interval、breakable prefill graph），因此不是 wheel 的单独 A/B；c24 在取消 HiCache 后未重跑。加入补丁 08 后 c12 的吞吐 +0.1%、p90 interactivity -0.7%，在运行间噪声范围内：PR 测得的收益（12 个请求时 eager 模式下每步约 2 ms）在本次 replay 中不可见，日志也无法显示稀疏路径是否被执行。c48 没有可比设置的更早运行。此前 c1 使用 admission 2（2x CONC）的运行为 13,467 tok/s/GPU、p90 175.2，因此 c1、c2 保持 admission = CONC。以上均为单次本地运行，不是 CI 数据。此设置下尚未运行：c2、c24、c48、c80、GSM8K 与 e2e Test workflow。
AI 模型使用说明