Repository navigation
[AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: extend baseline to conc 24/48/80 with HiCache DRAM offload, no patches / [AMD][MI35X] Qwen3.8-Flash-Next：基线扩展至并发 24/48/80（HiCache DRAM 卸载，无补丁） - #4779
ChangLiu0709 merged 17 commits into
Conversation
Same image and recipe as the merged round 02 baseline (#4542) for c1-c16. Add a HiCache DRAM-offload search-space entry for c24, c48 and c80 with the server settings of the AMD submission recipes (FP8 KV, mem-fraction-static 0.88, extra_buffer_lazy Mamba pool, hicache-size 200/298), without the extra AITER/SGLang patches those recipes carry.
Co-authored-by: Cursor <cursoragent@cursor.com>
…e c24/c48/c80 entry to PR 4779 Co-authored-by: Cursor <cursoragent@cursor.com>
…c24-c48-c80 Co-authored-by: Cursor <cursoragent@cursor.com>
…ry after the main merge Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Nikolai <289648307+amd-nprotaso@users.noreply.github.com> Co-authored-by: Christian Gilli <272772515+amd-cgilli@users.noreply.github.com> Co-authored-by: Amanzhol Salykov <asalykov@amd.com> Co-authored-by: wufann <36477220+wufann@users.noreply.github.com> Co-authored-by: Thomas Wang <1am9trash@gmail.com> Co-authored-by: Chun-Hung Yeh <Chun-Hung.Yeh@amd.com> Co-authored-by: jacky.cheng <yichiche@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chun Fang <chun.fang@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @ChangLiu0709's task in 27s —— View job ReviewLGTM - no blocking issues found.
I couldn't run |
ChangLiu0709
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # c24, c48 and c80 keep ReplaySSM and the PLE offload, store the target and draft KV | ||
| # caches in FP8 (E4M3), and add HiCache host-DRAM offload with a Mamba state pool | ||
| # that tracks a checkpoint every 256 tokens (extra_buffer_lazy). HiCache takes the | ||
| # point's 299 GB DRAM share from the matrix generator (hicache-size 200 at c24, 298 | ||
| # at c48 and c80); TOTAL_CPU_DRAM_GB records it. mem-fraction-static is 0.88, used | ||
| # as given. c24 admits 2x CONC; c48 and c80 admit 128 requests. | ||
| # c80 caches only the prompt on finish and keeps at most two Mamba states per path. |
There was a problem hiding this comment.
🟡 nit (optional): New header comments describe the overrides and config and restate values already in the YAML, which the mandatory comment rule forbids. sweep:^# (c24, c48 and c80|Concurrency 1-16 serve) Examples: agentic.yaml:171-177 repeats FP8 KV, hicache-size 200/298, mem-fraction-static 0.88 and admission 48/128. amd-master.yaml:259-260 summarises the config's concurrencies. Fix: delete both blocks. Keep only short line-level whys, such as the existing strip-thinking-cache and mamba-max-states-per-path comments.
Why this was flagged
AGENTS.md:56 says there must be no header comments describing a file, config, recipe, override or section. AGENTS.md:58 also says comments must never restate the code. The 7-line block at agentic.yaml:171-177 sits above override_c24 and describes all three overrides. It repeats values set a few lines below: kv-cache-dtype fp8_e4m3, hicache-size 200/298, mem-fraction-static 0.88, max-running-requests 48/128 and mamba-track-interval 256. The new lines at amd-master.yaml:259-260 ('Concurrency 1-16 serve from GPU KV; 24, 48 and 80 add an FP8 KV cache and HiCache host-DRAM offload.') are a config-level summary of the search-space rows. These comments drift out of date as soon as a value changes, and maintainers end up with prose that contradicts the code. Runtime behaviour is unaffected, so this is a nit.
Verification: nit. No runtime behavior breaks. AGENTS.md under "Comments (mandatory, every file)" says "No header comments describing a file, topology, config, recipe, override, or section" and "never restate the code". The diff adds a 7-line block at agentic.yaml:171-177, above override_c24:, that restates values set below it, and amd-master.yaml:259-260, which summarize the search-space rows.
chunfangamd
added
the
all-evals
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41882345112 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41882345112 |
|
/use 41882345112 |
|
@ChangLiu0709 staged run 41882345112. This run remains available across future |
…ry at the end after the branch update
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41882345112
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41882345112
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Evals: run 41882345112 ran GSM8K at all nine points (
all-evals) on the PR image: strict 0.9788 (c1), 0.9795 (c2), 0.9795 (c4), 0.9773 (c8), 0.9773 (c12), 0.9773 (c16), 0.9803 (c24), 0.9773 (c48) and 0.9780 (c80), n = 1319, SE about 0.004. No row trails another by more than 2σ, and all are within 2σ of the ingested qwen3.8next peers (MI355X FP4 0.9666–0.9795, B300 FP4 0.9765–0.9780). Each eval server ran its own point's path:- c1–c8 built the monokernel (
qwen38_flash_next_hipandqwen38_flash_next_mtp_draft_hip). - c12 and c16 ran ReplaySSM with PLE offload.
- c24–c80 allocated FP8 E4M3 target and draft KV pools and attached HiCache (
hicache_attached=True, plus host KV and Mamba pools). They ran withSGLANG_USE_AITER_UNIFIED_ATTN=1, which makes the aiter backend use unified attention for decode and target verify whenever it is set. c80 also ranstrip_thinking_cacheandmamba_max_states_per_path=2. - FP8 KV is the only new accuracy lever, and its three points show no drop.
- c1–c8 built the monokernel (
- Cookbook: sgl-project/sglang#48432 merged.
- Its Concurrency 24, 48 and 80 commands match every effective server arg and env var of the c24, c48 and c80 servers. That covers FP8 E4M3 target and draft KV,
SGLANG_USE_AITER_UNIFIED_ATTN, mem fraction 0.88, admission 48/128/128, the Mamba cache settings and HiCache (200/298 GB, kernel I/O,page_first,write_back). At c80 it also covers--strip-thinking-cacheand--mamba-max-states-per-path 2. - c1–c16 are unchanged from #4542 and documented by sgl-project/sglang#47951. Their published commands still pass
--default-chat-template-kwargs '{"enable_thinking": false}', a difference carried over from #4542. This PR's servers all run the thinking-on template (default_chat_template_kwargs: None), as InferenceX requires: srt-slurm AgentX points runTHINKING_MODE=thinking_on, the curve the injected 2.32 comes from. - Excluded as InferenceX-only or inert:
enable-metrics,enable-cache-reportand the benchmark env (KV_OFFLOADING,TOTAL_CPU_DRAM_GB,CONC);PYTHONNOUSERSITEandSGLANG_TIMEOUT_KEEP_ALIVE;ROCM_QUICK_REDUCE_QUANTIZATION=NONE, because TP1 runs no all-reduce;SGLANG_ENABLE_SPEC_V2=1, which SGLange9c18259does not read;SGLANG_USE_AITER=1, the image default.
- Its Concurrency 24, 48 and 80 commands match every effective server arg and env var of the c24, c48 and c80 servers. That covers FP8 E4M3 target and draft KV,
- Golden AL: all nine points inject
SGLANG_SIMULATE_ACC_LEN=2.32; the decode logs average 2.32 at c1 and c80. On main146a0b64,signoff_facts golden-alselectsqwen3.8-flash-next-fp8__mtp__native__default-default.yaml(thinking_on, 3 draft tokens), which gives 2.32 and matches. The AL gate ran on the c80 server (FP8 KV, HiCache) and measured a natural AL of 2.5407 on SPEED-Bench coding. That is 1.095× the golden value (above, report-only). - Draft precision: the MTP head (
mtp.*ofamd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8@bd6e49d9) runs as shipped.- Shipped: W8A8 FP8 attention and indexer projections, W4A4 MXFP4 routed and shared experts, and BF16 router, HC mixers,
fc_embedding/fc_hiddenand norms. The head loads with the checkpoint's own Quark config. - c1–c16: unchanged from #4542. The draft KV is BF16, and at c1–c8 the monokernel reads the FP8 and MXFP4 weights in place and keeps the per-token FP8 activations.
- c24–c80: the draft KV is FP8 E4M3 because the target KV is.
--speculative-draft-kv-cache-dtype fp8_e4m3restates SGLang's default, which follows--kv-cache-dtype, and both pools allocatetorch.float8_e4m3fn. No draft weight or activation dtype changes. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis unset, and integer all-reduce is off (ROCM_QUICK_REDUCE_QUANTIZATION=NONE, TP1).
- Shipped: W8A8 FP8 attention and indexer projections, W4A4 MXFP4 routed and shared experts, and BF16 router, HC mixers,
- Changelog: the entry covers the image, the unchanged c1–c16 points, the 299 GB HiCache entry, FP8 target and draft KV,
SGLANG_USE_AITER_UNIFIED_ATTN, the Mamba settings, admission, the c80 cache settings and the absence of patches. - Validation / reuse: run 41882345112 (attempt 1) on
72ac9f4: all nine agentic points, nine evals and the AL gate are green, and/use 41882345112is posted. Requests dropped withInvalidInferenceResultError: 1 at c8, 2 at c12, 1 at c16, 4 at c24, 9 at c48 and 13 at c80 (at most 0.2%), about the same rate as #4542's run. The evidence is still current under Check 1(b2) and (b3):-
The later commits (
e7ad07cd7,5d89c7d03,ce13f213b,5e957ca1d) only merge main and re-append this PR's changelog entry. The recipe, the master entry and the changelog entry are identical to72ac9f4. -
infx.matrix.plan --all-evalsgenerates the same 18 entries (nine points and nine evals), field for field, at72ac9f4and at5e957ca1d. -
srt-slurm v2.66.2 (#4784) is a single TokenSpeed commit, and only its TokenSpeed launch snapshots change.
-
The other main changes on this path are:
- client metadata (
SPEC_ALGORITHM/SPEC_DRAFT_MODEL, #4805); - curve selection that still gives 2.32 (#4661, #4663);
- the report-only AL gate, now run on the eval server (#4810);
- a wait for prior-job VRAM before setup (#4813, #4814, #4827);
- RAPL CPU-power telemetry on mi355x-amds (#4154).
None of them changes a server argument, environment variable, client workload or eval.
- client metadata (
-
- Images: unchanged from #4542.
lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031@sha256:3867dc07…is built from SGLangmainate9c18259, with aiter285579b6, sgl-amd-monob10f3b71and morif39fa765, each on its default branch. - Other: no patches or setup scripts.
--strip-thinking-cache(cache only the prompt of a finished reasoning request) and--mamba-max-states-per-pathare upstream cache policies: they change cache contents, not computation. Noappend-only; Checks 16 and 17 are N/A. - Pareto (p90 throughput per GPU vs E2EL, via
pareto_coverage): one curve, made of c1–c16 on GPU KV and c24–c80 on HiCache, from the ninebmk_agentic_*artifacts of run 41882345112. 9 measured, 0 invalid, 7 on the frontier (c2, c8, c12, c16, c24, c48, c80): PASS. c1 is dominated by c2, and c4 by c8.
Signed: @chunfangamd
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep + evals): PASS — run 41882345112 (attempt 1) ran on in-PR commit ✅ Check 2 (Evals): PASS — GSM8K ran on the PR image ( ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#48432 merged 2026-11-04, and its c24/c48/c80 commands match every production setting in the effective recipe. That includes FP8 target and draft KV, mem-fraction 0.88, admission and CUDA-graph sizes, the Mamba settings, the HiCache flags, ✅ Check 4 (Reuse command): PASS — COLLABORATOR ✅ Check 5 (Checklist template): PASS — all 20 items in the current ✅ Check 6 (Upstream images / provenance / ordering): PASS — the image is upstream ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks / integer all-reduce): PASS — there is no FLOP-reducing override; ✅ Check 9 (Chat template): PASS — the base benchmark env sets ✅ Check 10 (Engine patches): PASS — no reachable engine patch or setup script. The only ✅ Check 11 (Golden AL): PASS — the c80 log shows ➖ Check 12 (Append-only): N/A — the new changelog entry has no ✅ Check 13 (Draft precision): PASS — the embedded MTP head of ✅ Check 14 (Pareto coverage): PASS — one curve (qwen3.8next / agentic-coding / mi355x_sglang / fp4 / p90 E2EL vs tput_per_gpu) built from the nine ✅ Check 15 (PR description): PASS — the body's throughput, interactivity, TTFT, GSM8K, error-drop counts, 299 GB share, validation SHA and later-commit claims all match the assessed code and artifacts. ➖ Check 16 (NVIDIA overview): N/A — MI355X only, so no NVIDIA configurations are affected. ➖ Check 17 (Metrics publication): N/A — this is an SGLang serving path, not Dynamo TRT-LLM. ✅ Check 18 (Recipe/image/sweep consistency): PASS — the Assessed commit: |
The Update-branch merges of main inserted the entry in front of the VideoGenX H3 entry, which cost that entry its config-keys header and made infx.matrix.plan reject the diff. Main's file is restored unchanged and the entry is appended after it.
…last The previous fix started from a stale origin/main and dropped newer main entries.
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 10, rawBenchmarkRows 18, coveredByPerJobFiles 9, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 11871, gpuMetricSamplesNew 43060, powerPublicationPoints 9, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Summary
Extends the merged Qwen3.8-Flash-Next FP4 MI355X SGLang AgentX baseline (#4542) from concurrency 1/2/4/8/12/16 to also cover 24, 48 and 80. The image, model and recipe for c1–c16 are unchanged:
lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031@sha256:3867dc07ba2530f914e1f210ed829f7a572a75286c8c362dbc22cbecf9b5a7e6, TP1/EP1, MTP. There is no custom image, no AITER/SGLang patch and no setup script.qwen3.8next-fp4-mi355x-sglang-agentic-mtp: one new search-space entry,kv-offloading: dramwith thehicachebackend,conc-list: [24, 48, 80], sameagentic.yamlrecipe. Each point takes the 299 GB DRAM share ofdram-utilization: 0.80.override_c24,override_c48andoverride_c80: FP8 E4M3 target and draft KV cache,mem-fraction-static0.88 (SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1),SGLANG_USE_AITER_UNIFIED_ATTN=1, ReplaySSM and PLE offload kept,extra_buffer_lazyMamba pool (track interval 256,max-mamba-cache-size960 / 720 / 640), HiCache (kernelIO,page_first,write_back,hicache-size200 at c24 and 298 at c48 and c80).strip-thinking-cacheandmamba-max-states-per-path 2.inferencex-e2e/perf-changelog.yaml.Cookbook coverage
The production serving commands for the new points (c24, c48, c80: FP8 E4M3 KV cache, HiCache host-memory offload, the
extra_buffer_lazyMamba pool, batch limits) are documented in the SGLang cookbook PR sgl-project/sglang#48432, merged on 2026-11-04. Concurrency 1–16 are already covered by the merged round 02 cookbook PR sgl-project/sglang#47951 (see #4542). The new commands were compared with the recipe's renderedargsandenvat each point: 0 missing, 0 conflicting, 0 extra settings besides--hostand--port; telemetry flags and benchmark-only settings are excluded, as for the earlier points.Validation
PR Run Sweep 41882345112 (attempt 1, head
72ac9f4, withall-evals) passed at all nine points and ran a GSM8K eval at every point, and/use 41882345112is posted. The later commits only merge main and re-append the changelog entry; the recipe and config are unchanged. The job logs show only the runner-side573-participating-gpus.patchon the srt-slurm launcher checkout and no patch inside the container. An earlier branch e2e dispatch, 41834528561 (head2765910fa), gave matching numbers.GSM8K (1319 questions, strict match): c1 0.9788, c2 0.9795, c4 0.9795, c8 0.9773, c12 0.9773, c16 0.9773, c24 0.9803, c48 0.9773, c80 0.9780.
Known gaps:
records_error_dropped, allInvalidInferenceResultError), 9 of 5,227 at c48 and 13 of 7,603 at c80 being the largest counts. The cause is not yet investigated; [AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: 20261031 image, monokernel at c1-c8 / [AMD][MI35X] Qwen3.8-Flash-Next：20261031 镜像，c1-c8 启用 monokernel #4542's sweep shows the same rate at c2–c16.AI model disclosure
PR checklist and change type
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
摘要
将已合并的 Qwen3.8-Flash-Next FP4 MI355X SGLang AgentX 基线（#4542）从并发 1/2/4/8/12/16 扩展到并发 24、48、80。c1–c16 的镜像、模型和 recipe 均不变：
lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031@sha256:3867dc07…，TP1/EP1，MTP。不使用自定义镜像、不打 AITER/SGLang 补丁，也没有 setup 脚本。qwen3.8next-fp4-mi355x-sglang-agentic-mtp：新增一个搜索空间条目，kv-offloading: dram，后端hicache，conc-list: [24, 48, 80]，沿用同一个agentic.yaml。每个点使用dram-utilization: 0.80对应的 299 GB DRAM 份额。override_c24、override_c48、override_c80：目标与草稿 KV cache 使用 FP8 E4M3，mem-fraction-static为 0.88，保留 ReplaySSM 与 PLE offload，采用extra_buffer_lazyMamba 池，并启用 HiCache（c24 的hicache-size为 200，c48 与 c80 为 298）。strip-thinking-cache与mamba-max-states-per-path 2。inferencex-e2e/perf-changelog.yaml末尾追加一条记录。Cookbook 覆盖
新增并发点（c24、c48、c80：FP8 E4M3 KV cache、HiCache 主机内存卸载、
extra_buffer_lazyMamba 池、批量上限）的生产服务命令记录在 SGLang cookbook PR sgl-project/sglang#48432，该 PR 已于 2026-11-04 合入。并发 1–16 已由已合入的第 02 轮 cookbook PR sgl-project/sglang#47951 覆盖（对应 #4542）。新命令已与配方在各点渲染出的args和env逐项比对：无缺失、无冲突、除--host与--port外无多余设置；遥测标志与仅基准用的设置不收录，与此前各点一致。验证
PR Run Sweep 41882345112（attempt 1，head
72ac9f4，带all-evals）九个点全部通过，并在每个点都运行了 GSM8K 评测，结果见上方英文表格；/use 41882345112已发布。之后的提交只合并 main 并重新追加 changelog 条目，recipe 与配置均未改动。作业日志中只有 runner 侧对 srt-slurm 启动器的573-participating-gpus.patch，容器内没有任何补丁。此前的分支 e2e 触发 41834528561（head2765910fa）结果与之一致。GSM8K（1319 题，strict match）：c1 0.9788，c2 0.9795，c4 0.9795，c8 0.9773，c12 0.9773，c16 0.9773，c24 0.9803，c48 0.9773，c80 0.9780。
已知问题：
InvalidInferenceResultError），其中 c48 为 5,227 个中的 9 个，c80 为 7,603 个中的 13 个，原因尚未排查；[AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: 20261031 image, monokernel at c1-c8 / [AMD][MI35X] Qwen3.8-Flash-Next：20261031 镜像，c1-c8 启用 monokernel #4542 的 sweep 在 c2–c16 也有相同比例。AI 模型使用说明