Repository navigation
perf(amd): Kimi-K3.1 MI355X vLLM agentic: per-conc max-num-seqs, conc 1 DCP1, conc 4 seven draft tokens / perf(amd): Kimi-K3.1 MI355X vLLM agentic：按并发调整 max-num-seqs，并发 1 改为 DCP1，并发 4 使用 7 个草稿 token - #4887
Conversation
… 1 DCP1, conc 4 seven draft tokens Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I checked two other things and both are fine. The DSpark probabilistic-block golden AL file has a 7-token entry (3.69), which matches the changelog. The null values in override_gpu_kv_c1 do remove the DCP args, because the srtctl merge in infx/srt_slurm/variants.py deletes keys set to null.
Extended reasoning...
The PR changes the Kimi-K3.1 MI355X vLLM agentic recipe overrides, the matching conc 1 dcp-size row in amd-master.yaml, and adds a changelog entry; it touches no security-sensitive code. The note records two checks beyond the inline finding: the golden AL curve covers 7 draft tokens, and null overrides delete the DCP keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hyukjlee
added
the
non-canary-full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42371416815 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42371416815 |
|
/reuse-sweep-run 42371416815 |
- The image stays vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336). - Concurrency 1 moves from TP8 x DCP8 to TP8 x DCP1 (master config dcp-size 8 -> 1; the recipe's c1 override drops decode-context-parallel-size, dcp-comm-backend and cp-kv-cache-interleave-size). It keeps GPU-resident KV, 7 draft tokens and the latency_specialized decode backend. Concurrency 4-48 stay TP8 x DCP8 with DRAM offload. - Concurrency 4: DSpark draft tokens 5 -> 7, so throughput runs use golden acceptance length 3.69 instead of 3.41. The decode capture sizes change from 6, 12 to 8, 16 to match max-num-seqs 2 x (1 + 7). - max-num-seqs at concurrency 10/16/24/32 changes from 6/10/16/24 to 7/8/13/20. Concurrency 10 adds capture size 42 (7 x (1 + 5)). Otherwise the draft lengths, capture lists, max-num-batched-tokens and gpu-memory-utilization at these points, and every setting at concurrency 48, are unchanged. - No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config names the draft and sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache stays FP8, the same as the target's. DCP1 at concurrency 1 and 7 draft tokens at concurrency 4 do not change these types. - 镜像保持为 vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1（vLLM 0.0.0+mum.14d1ec2336）。 - 并发 1 从 TP8 x DCP8 改为 TP8 x DCP1（master config 的 dcp-size 8 -> 1；recipe 的 c1 override 去掉 decode-context-parallel-size、dcp-comm-backend 和 cp-kv-cache-interleave-size）。它仍使用 GPU 常驻 KV、7 个草稿 token 和 latency_specialized decode 后端。并发 4-48 仍为 TP8 x DCP8 并使用 DRAM offload。 - 并发 4：DSpark 草稿 token 数 5 -> 7，因此吞吐测试使用 golden acceptance length 3.69 而不是 3.41。decode capture sizes 从 6、12 改为 8、16，以匹配 max-num-seqs 2 x (1 + 7)。 - 并发 10/16/24/32 的 max-num-seqs 从 6/10/16/24 改为 7/8/13/20。并发 10 新增 capture size 42（7 x (1 + 5)）。除此之外，这些点的草稿长度、capture 列表、max-num-batched-tokens 和 gpu-memory-utilization，以及并发 48 的所有设置均不变。 - DSpark 草稿精度无变化：Inferact/Kimi-K3.1-Pro-DSpark-preview（revision 6a2ef20a）发布的权重全部为 BF16 且没有 quantization_config，speculative config 指定了草稿且未设置量化，因此 get_draft_quant_config() 返回 None，K3DSparkModel 以 BF16 加载各层，并使用目标模型的 BF16 embedding 和 LM head。其 KV cache 保持 FP8，与目标模型相同。并发 1 的 DCP1 和并发 4 的 7 个草稿 token 都不改变这些类型。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
@seungrokj update vllm recipes plz and then merge it
seungrokj
changed the title
|
Draft data types: no change. The DSpark draft
The image is unchanged from #4806, so the framework is the same: vLLM
The recipe sets no 中文草稿数据类型：无变化。 DSpark 草稿
镜像与 #4806 相同，因此框架也相同：vLLM
本 recipe 在任何点都没有设置 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe vllm-project/vllm#66343
- #4887 (comment)
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj Blocking: the linked upstream recipe change is not merged, so Check 3 fails. Merge it (or link an already-published recipe page that covers every affected setting) before this PR merges. ❌ Check 3 (Upstream recipe): FAIL — Linked upstream recipe is not merged: vllm-project/vllm#66343 is
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep + evals): PASS — In-PR commit ✅ Check 2 (Eval accuracy): PASS — gsm8k em_strict is 0.9795 at c1 (TP8/DCP1, the changed latency-specialized path) and 0.9826 at c48, both on the PR image. The gap to the sibling is 0.003 (threshold 0.011) and to the #4806 cohort (run 42193891273) is 0.0008/−0.002, so neither row is materially weaker. ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — All 19 current template items are present and checked. ✅ Check 6 (Upstream images): PASS — ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No arch hacks): PASS — There are no ✅ Check 9 (Chat template): PASS — DSpark points run the AgentX chat-trace replay ( ✅ Check 10 (No engine patches): PASS — The recipe and diff have no patches, setup scripts, source edits or wheel installs. ✅ Check 11 (Golden AL): PASS — 7 draft tokens (c1, c4) map to golden AL 3.69 on curve ➖ Check 12 (Append-only): N/A — The new changelog entry is not ➖ Check 15 (PR description): N/A — The body's technical claims (per-conc changes, image, run 42371416815 at ➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X. ➖ Check 17 (Metrics publication): N/A — This is a vLLM aggregated path with no Dynamo TRT-LLM settings. ✅ Check 18 (Recipe consistency): PASS — Master Assessed commit: |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 7, rawBenchmarkRows 7, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 321080, powerPublicationPoints 7, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Kimi-K3.1-Pro MI355X vLLM AgentX: per-concurrency retune on the same image as #4806 (
vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10…). Concurrency 48 is unchanged.The PR's own sweep regresses concurrency 10 by about 8 % in throughput (run 42371416815, both c10 jobs), although the local A/B below showed +1.3 %. All other points are flat or better. See the table under Validation.
Changes
dcp-size: 1; the c1 override drops the DCP args)Draft data types: no change.
Inferact/Kimi-K3.1-Pro-DSpark-preview(revision6a2ef20a) ships all-BF16 weights with noquantization_config. The recipe sets no quantization, so the draft runs as shipped: BF16 weights and activations, the target's BF16 embedding and LM head, and an FP8 KV cache, the same as the target's. Evidence is in the PR comment.Validation
PR sweep run 42371416815 at
e9a33875(same recipe and master config as the head): every benchmark and eval job passed. Compared with #4806's sweep (run 42193891273); interactivity is 1 / P90 TPOT:Validation details: local A/B (no run links)
Local MI355X, 3600 s AgentX, synthetic golden AL, against #4806's sweep:
Each change first won a same-node 1800 s A/B against the #4806 config. At conc 48, max-num-seqs 40 was neutral over 3600 s (−0.4 % tput / +2.1 % P90), so conc 48 is unchanged.
中文
Kimi-K3.1-Pro MI355X vLLM AgentX：在与 #4806 相同的镜像（
vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10…）上按并发度重新调优。并发 48 不变。本 PR 自身的 sweep 中，并发 10 的吞吐下降约 8 %（run 42371416815，两个 c10 作业均如此），而下方本地 A/B 显示为 +1.3 %。其余各点持平或提升，见"验证"中的表格。
变更
dcp-size: 1；c1 override 去掉 DCP 参数）草稿数据类型：无变化。
Inferact/Kimi-K3.1-Pro-DSpark-preview（revision6a2ef20a）发布的权重全部为 BF16，且没有quantization_config。本 recipe 未设置任何量化，因此草稿按发布形式运行：BF16 权重和激活，使用目标模型的 BF16 embedding 和 LM head，KV cache 为 FP8，与目标模型相同。证据见 PR 评论。验证
PR sweep run 42371416815 在
e9a33875（recipe 和 master config 与 head 相同）上运行，所有 benchmark 和 eval 作业均通过。与 #4806 的 sweep（run 42193891273）对比；interactivity 为 1 / P90 TPOT：本地 A/B（无运行链接）：本地 MI355X、3600 s AgentX、synthetic golden AL，与 #4806 的 sweep 对比：
每项变更都先在同一节点上以 1800 s A/B 胜过 #4806 的配置。在并发 48，max-num-seqs 40 在 3600 s 下持平（吞吐 −0.4 % / P90 +2.1 %），因此并发 48 保持不变。
AI model disclosure
claude-opus-5-5(Claude Code): rewrote the perf-changelog block from the diff, verified the draft data types at vLLM14d1ec23363aand in the draft's safetensors header, added the sweep comparison, and revised this description.🤖 Generated with Claude Code