Repository navigation
Conversation
Add kimik3-fp4-mi355x-atom-disagg-agentic: six Kimi-K3 MXFP4 AgentX points on MI355X with ATOM prefill/decode behind AToMesh and Mooncake P->D transfer. 1P1D TP8 DSpark7 c1, 1P2D TP8+DCP8 DSpark7 c16, 1P1D TP8+DCP8 DSpark4 c48/c72/c88 and 1P2D TP8+DCP8 DSpark4 c48. The DCP8 prefill keeps paged KV and KDA state images in an LMCache CPU tier; the DSpark draft stays BF16. 新增 kimik3-fp4-mi355x-atom-disagg-agentic：MI355X 上 6 个 Kimi-K3 MXFP4 AgentX 测试点，ATOM 预填充/解码分离，经 AToMesh 路由，Mooncake 负责 P->D 传输。包括 1P1D TP8 DSpark7 c1、1P2D TP8+DCP8 DSpark7 c16、1P1D TP8+DCP8 DSpark4 c48/c72/c88 和 1P2D TP8+DCP8 DSpark4 c48。DCP8 预填充 把分页 KV 和 KDA 状态镜像放在 LMCache CPU 层；DSpark 草稿模型保持 BF16。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
为 Kimi-K3 ATOM P/D 条目填入 #4311 链接。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| MC_GID_INDEX: '@fabric.rdma-gid-index' | ||
| NCCL_IB_DISABLE: '1' |
There was a problem hiding this comment.
🔴 All six points may see Mooncake prefill-to-decode KV transfers fail or stall if mi355x-amds rails are isolated. The ATOM disagg recipe already on this runner relies on ATOM_MOONCAKE_MATCHED_RAILS: auto. The shared &env block sets MC_GID_INDEX and NCCL_IB_DISABLE but leaves matched rails out. Without it, Mooncake falls back to dest-device affinity, which the sibling says rail isolation breaks. Fix: add ATOM_MOONCAKE_MATCHED_RAILS: auto to the &env anchor so prefill and decode both get it, or note why Kimi-K3 does not need it.
Why this was flagged
The condition is that mi355x-amds is rail-isolated, as the sibling recipe on the same runner states. Its comment at dsv4/atom/mi355x-fp4/agentx/disagg-lmcache-dspark.yaml:33-35 reads "Rail-isolated RDMA: matched rails replace dest-device affinity." and it sets ATOM_MOONCAKE_MATCHED_RAILS: auto. Both recipes use runner cluster:mi355x-amds, framework: atom-disagg and kv-p2p-transfer: mooncake. The new &env anchor (disagg-variants.yaml:40-63) copies that recipe's MC_GID_INDEX: '@ fabric.rdma-gid-index' and NCCL_IB_DISABLE: '1' lines but drops the matched-rails line. Both roles use this anchor, since decode takes env: *env. No override adds the setting back. The runners.yaml fabric entry for mi355x-amds (lines 810-814) has no rail flag that would supply it. So a prefill rank may post RDMA writes from a NIC that cannot reach the decode NIC. KV handoffs on every point would then error or hang, and the sweep would fail. The PR description does not mention this setting.
Verification: The new shared env: &env anchor (kimik3/atom/mi355x-fp4/agentx/disagg-variants.yaml lines 40-63) has no ATOM_MOONCAKE_MATCHED_RAILS. Decode uses env: *env (line 88), so both roles lack the setting. The sibling recipe is dsv4/atom/mi355x-fp4/agentx/disagg-lmcache-dspark.yaml. The new recipe kept lines 34 and 36 of its lines 33-36 and dropped line 35, ATOM_MOONCAKE_MATCHED_RAILS: auto.
Jasen2201
added
the
full-sweep-fail-fast
Keep the Kimi-K3 ATOM P/D changelog entry at the physical end after main's new entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201
added
full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41377208565 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41377208565 |
Keep the Kimi-K3 ATOM P/D changelog entry at the physical end after #4295. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ugepages On mi355x-amds a 4 KiB LMCache pool registered with hipHostRegister stays movable. Each page the kernel migrates (compaction, reclaim) stalls the rank's GPU queues until the driver re-validates the pool, and GLM-5.2 ATOM prefills with ~1 TiB pools hung in the post-pool barrier on 3 of 14 timed starts there. The K3 DCP8 prefill pins 8 x 224 GiB (8 x 128 GiB at c16). Set LMCACHE_LOCAL_CPU_USE_HUGEPAGES on the prefill so LMCache maps each pool with MAP_HUGETLB 2 MiB pages bound to its NUMA node. host_setup reserves the pages on the head node (the prefill's) through the cluster's hugepage_sanity.sh, checks each NUMA node's share and that 400 GiB of other memory stays free for weight loading, and teardown releases them. The c1 point has no LMCache and is unchanged. 把 Kimi-K3 ATOM prefill 的 LMCache 池改用 2 MiB 大页，host_setup 在 prefill 节点预留、teardown 释放，避免 4 KiB 池页面迁移导致启动 barrier 卡死。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201
added
non-canary-full-sweep-enabled
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
before sweeping, u need to fix the mooncake kvcache counters for atomesh. chun and seungrok are working on a code change to fix it iirc. tho the MVP code change needs some serious changes to make it high quality
functionstackx
removed
the
non-canary-full-sweep-enabled
|
oh nvm, this is Mooncake for RDMA & LMCache for kvcache, chun & @seungrokj does lmcache atomesh with mooncake rdma have this issue or is it mooncake store that has this issue? |
…tom-disagg-agentx # Conflicts: # inferencex-e2e/perf-changelog.yaml
Jasen2201
added
the
full-sweep-fail-fast
Jasen2201
added
full-sweep-enabled
|
/reuse-sweep-run 40782441830 |
Resolve the perf-changelog conflict with main's new entries (#4647, #4627, #4596, #4526, #4664) using prepare_perf_changelog_merge resolve-conflict: main's bytes are kept and this PR's entry is re-appended unchanged. The recipe, the master entry and the changelog entry are byte-identical to the swept commit dd7a1f1 (run 40782441830, pinned by /reuse-sweep-run). 用 prepare_perf_changelog_merge resolve-conflict 解决与 main 新条目 （#4647、#4627、#4596、#4526、#4664）的 perf-changelog 冲突：保留 main 的原始字节，本 PR 的条目原样追加在末尾。recipe、master 条目和 changelog 条目与 sweep 所用提交 dd7a1f1（run 40782441830，已由 /reuse-sweep-run 固定）逐字节一致。 [skip-sweep] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/ 依据 diff 重写 Kimi-K3 MI355X ATOM P/D 条目 - New config on rocm/atom-dev:nightly_202610190554 (ROCm/ATOM daa4f1bb8), pinned by digest; the AToMesh router is pinned to the same ATOM commit. - Add six P/D points with Mooncake P->D transfer: 1P1D TP8 DSpark7 c1, 1P2D TP8+DCP8 DSpark7 c16, 1P1D TP8+DCP8 DSpark4 c48/c72/c88 and 1P2D TP8+DCP8 DSpark4 c48. - Both roles: FP8 KV, block 128, ptpc_fp8 online quantization with the MXFP4 routed experts excluded, FlyDSL FP8 prefill attention, prompt-end state checkpoint only; admission 2x conc (32 at c1); decode FULL graphs up to 16/256/480/720/880. - Prefill LMCache CPU tier: 128 GB per rank at c16, 224 GB per rank with state images capped at 128 GB and on-demand checkpoints at the DSpark4 points; c1 GPU-resident, no decode CPU tier; ReplaySSM at DCP8; AITER_REUSE_IDENTICAL_COMM_GROUPS=1 at c72/c88. - Prefill LMCache pools backed by 2 MiB HugeTLB pages reserved by host_setup and released by teardown. - AToMesh routes with policy random. - ATOM daa4f1bb8 includes ROCm/ATOM#2449 (evicted KDA state images are no longer advertised). - Draft: Inferact/Kimi-K3-DSpark runs as shipped in BF16 and caches in the target's FP8 MLA KV pool. - 新配置使用 rocm/atom-dev:nightly_202610190554（ROCm/ATOM daa4f1bb8），按 digest 固定；AToMesh 路由固定在同一 ATOM 提交。 - 新增 6 个使用 Mooncake P->D 传输的 P/D 测试点：1P1D TP8 DSpark7 c1、1P2D TP8+DCP8 DSpark7 c16、1P1D TP8+DCP8 DSpark4 c48/c72/c88 和 1P2D TP8+DCP8 DSpark4 c48。 - 两个角色：FP8 KV、block 128、ptpc_fp8 在线量化（排除 MXFP4 路由专家）、FlyDSL FP8 预填充注意力，仅保留 prompt 末尾状态 checkpoint；准入为 2 倍并发（c1 为 32）；解码 FULL 图捕获至 16/256/480/720/880。 - 预填充 LMCache CPU 层：c16 每 rank 128 GB，DSpark4 测试点每 rank 224 GB，状态镜像上限 128 GB 并按需补充 checkpoint；c1 完全驻留 GPU，解码无 CPU 层；DCP8 测试点开启 ReplaySSM；c72/c88 设置 AITER_REUSE_IDENTICAL_COMM_GROUPS=1。 - 预填充 LMCache 池使用 2 MiB HugeTLB 页，由 host_setup 预留、teardown 释放。 - AToMesh 使用 random 策略路由。 - ATOM daa4f1bb8 包含 ROCm/ATOM#2449（不再公布已驱逐的 KDA 状态镜像）。 - 草稿：Inferact/Kimi-K3-DSpark 按发布时的 BF16 运行，KV 写入目标的 FP8 MLA KV 池。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Draft data types: no change. The DSpark draft (
At ATOM
So the draft runs as shipped: BF16 weights and activations for every draft layer, with the FP8 MLA KV cache shared with the target. This is a new config key, so there is no earlier state of this config to compare against. The ATOM CI runs cited in the PR body quantize the draft to FP8; this recipe does not. 中文草稿数据类型：无变化。 DSpark 草稿（
在 ATOM
因此草稿按发布状态运行：所有草稿层的权重和激活均为 BF16，KV cache 与目标共用 FP8 MLA 池。这是新的配置键，没有可对比的旧状态。PR 描述中引用的 ATOM CI 运行把草稿量化为 FP8，本配方不这样做。 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe at https://github.com/ROCm/ATOM/blob/main/recipes/mesh/Agentic-Kimi-K3.md
- #4311 (comment)
- bypass
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj Blocking: the c1 Kimi-K3 tool-call eval is materially weaker than its sibling topology in the same run, and the sign-off has no rerun or explanation for it. The new perf-changelog entry also omits the INT4 quick-reduce all-reduce quantization that both roles enable. ❌ Check 2 (Evals pass): FAIL — Materially weaker eval with no rerun-backed explanation. In run 40782441830 (in-PR commit dd7a1f1, per ❌ Check 18 (Recipe/image/sweep consistency): FAIL — (f2) The new
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep + evals): PASS — In-PR commit dd7a1f1 has green ➖ Check 3 (Upstream recipe): N/A — Only ✅ Check 4 (Reuse command): PASS — COLLABORATOR ✅ Check 5 (Latest checklist): PASS — All 20 items in the current PR_REVIEW_CHECKLIST.md template are present and checked. ✅ Check 6 (Upstream images / ordering / provenance): PASS — (a) N/A: there are no vLLM/SGLang entries. (b) Kimi-K3 on ✅ Check 7 (Deprecated models): PASS — Kimi-K3 ( ✅ Check 8 (No architecture hacks): PASS — (a) There are no hf-overrides or model-file edits. (b) Evals cover ✅ Check 9 (Chat template for spec decode): PASS — The benchmark env sets ✅ Check 10 (No engine patches): PASS — There is no reachable patching. ✅ Check 11 (Golden AL): PASS — For ➖ Check 12 (Append-only): N/A — The new changelog entry does not set ✅ Check 13 (Draft precision): PASS — ✅ Check 15 (PR description): PASS — The body's claims match the pinned code and run evidence: the six topologies, 2x conc admission, capture sizes, LMCache 128/224 GB with a 128 GB state cap, the digest pin, validation run 40782441830 at dd7a1f1, and the reuse comment. ➖ Check 16 (NVIDIA overview): N/A — Only MI355X (AMD) configurations are affected. ➖ Check 17 (Metrics publication): N/A — This is ATOM P/D behind AToMesh, with no Dynamo TRT-LLM path. Assessed commit: |
…(1 + draft) / 将 Kimi-K3 ATOM P/D 图捕获上限表述为 2 倍并发 x (1 + 草稿 token) - Decode captures every size up to 2x conc x (1 + draft tokens): 16 at c1, 256 at c16, 480 at c48, 720 at c72 and 880 at c88. Admission (max-num-seqs) is 2x conc except 32 at c1, so "admission x (1 + draft tokens)" overstated the c1 bound (256 instead of 16). - 解码捕获最多 2 倍并发 x (1 + 草稿 token 数) 的每个尺寸：c1 为 16、c16 为 256、c48 为 480、c72 为 720、c88 为 880。准入上限（max-num-seqs）为 2 倍并发，c1 例外为 32，因此"准入 x (1 + 草稿 token 数)"高估了 c1 的上限（256 而非 16）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe at https://github.com/ROCm/ATOM/blob/main/recipes/mesh/Agentic-Kimi-K3.md
- #4311 (comment)
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj Blocking: in the validation run, the c1 Kimi-K3 tool-call eval is materially weaker than its sibling topology, and the sign-off does not explain the gap with a rerun. The Check 18 changelog omission from the previous verdict is fixed at abe13e4. ❌ Check 2 (Evals pass): FAIL — A materially weaker eval has no explanation backed by a rerun. In run 40782441830 (in-PR commit dd7a1f1, attempt 4,
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — In-PR commit dd7a1f1 has green multi-node agentic, canary and agentic eval check-runs for all six points (run 40782441830). Later commits are four main merges and four changelog-only edits. The recipe blob (b145622) and the master entry are byte-identical at abe13e4. The merged launcher changes (DSV4/MiniMax-M3 retirement) and the srt-slurm bump ae3d687→59920ea (opt-in container reuse, ➖ Check 3 (Upstream recipe): N/A — Only an ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest template): PASS — Every item of the current ✅ Check 6 (Upstream images/ordering/provenance): PASS — (a) The PR adds an ATOM entry only, with no vLLM/SGLang entry. (b) Open-source engine entries already exist for kimik3 on ✅ Check 7 (Deprecated models): PASS — Kimi-K3 agentic coding is active in MODELS.md as of 2026-10-30. ✅ Check 8 (No arch hacks): PASS — (a) There are no architecture overrides. (b) ptpc_fp8 online quantization and ✅ Check 9 (Chat template): PASS — The AgentX benchmark sets ✅ Check 10 (No engine patches): PASS — There are no patches, setup scripts or wheel installs. ✅ Check 11 (Golden AL): PASS — ➖ Check 12 (Append-only): N/A — No new changelog entry sets ✅ Check 13 (Draft precision): PASS — ✅ Check 15 (PR description): PASS — The body's claims match abe13e4 and run 40782441830: topology, golden ALs, admission (2x conc, 32 at c1/c16), graph sizes (16/256/480/720/880), DRAM budgets (1794/1029 GB), image digest, validation run and reuse command. The ATOM CI table is labeled as an external reference. ➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X. ➖ Check 17 (Metrics publication): N/A — The ATOM framework does not use the Dynamo TRT-LLM metrics settings. ✅ Check 18 (Recipe/image/sweep consistency): PASS — (a) All six selectors generate. (b) Nodes, workers, TP/DCP, admission and DRAM budgets agree with master. (c) The digest-pinned image resolves ( Assessed commit: |
…P/D / 记录 Kimi-K3 ATOM P/D 使用 INT4 AITER quick all-reduce - Both roles run INT4-quantized AITER quick all-reduce (AITER_QUICK_REDUCE_QUANTIZATION=INT4) for the tensor-parallel all-reduces at every point; the shared recipe env sets it. - 两个角色在所有测试点上的张量并行 all-reduce 均使用 INT4 量化的 AITER quick all-reduce（AITER_QUICK_REDUCE_QUANTIZATION=INT4），由共享的 recipe 环境变量设置。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions / 记录 Kimi-K3 ATOM P/D 的 MoE 与 KDA 状态精度 - Both roles feed FP4 activations into the MXFP4 routed experts (AITER_SITUV2_A4W4=1; BF16 by default). - The MoE stage-2 reduce output is written as FP8 value/scale rows (AITER_FLYDSL_STAGE2_FP8=1). - The KDA recurrent state pool is stored in fp16 (ATOM_GDN_SSM_DTYPE=fp16; ATOM's default is fp32, and the recurrence still accumulates in fp32). - 两个角色向 MXFP4 路由专家输入 FP4 激活（AITER_SITUV2_A4W4=1；默认为 BF16）。 - MoE 第二阶段 reduce 的输出以 FP8 数值/缩放行写出（AITER_FLYDSL_STAGE2_FP8=1）。 - KDA 循环状态池以 fp16 存储（ATOM_GDN_SSM_DTYPE=fp16；ATOM 默认为 fp32，递推计算仍以 fp32 累加）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tom-disagg-agentx # Conflicts: # inferencex-e2e/perf-changelog.yaml
…0756 / 将 Kimi-K3 ATOM P/D 切换到 rocm/atom-dev:nightly_202610310756 - The image is rocm/atom-dev:nightly_202610310756 (ROCm/ATOM e281d58ac), pinned by digest; the AToMesh router pin follows it. - It carries ROCm/ATOM#2849, which forces Kimi-K3's tools channel on the GPU for tool_choice="required", the mode the Kimi tool-call eval sends. That eval scored 0.9338 at c1 on nightly_202610190554. - On this image the latent MoE projections (routed_expert_down_proj/up_proj) run in BF16, as the checkpoint stores them; daa4f1bb8 quantized them to MXFP4 at load. The changelog entry records it. - 镜像改为 rocm/atom-dev:nightly_202610310756（ROCm/ATOM e281d58ac），按 digest 固定；AToMesh 路由的固定版本随之更新。 - 该镜像包含 ROCm/ATOM#2849：对 tool_choice="required" 的请求，在 GPU 上强制进入 Kimi-K3 的 tools 通道，Kimi 工具调用评测发送的正是这种请求。在 nightly_202610190554 上，该评测的 c1 得分为 0.9338。 - 在该镜像上，潜空间 MoE 投影（routed_expert_down_proj/up_proj）按 checkpoint 存储的 BF16 运行；daa4f1bb8 会在加载时把它们量化为 MXFP4。changelog 条目已记录这一点。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jasen2201
added
full-sweep-enabled
functionstackx
removed
the
full-sweep-enabled
| state-checkpoint-interval-tokens: -1 | ||
| # Global ptpc_fp8 online quantization; the trailing entries keep the BF16 | ||
| # draft (unprefixed layers.* and context_proj) unquantized. | ||
| online_quant_config: '{"global_quant_config":"ptpc_fp8","exclude_layer":["lm_head","model.embed_tokens","*self_attn.[qkv]_conv1d*","*block_sparse_moe.experts*","*block_sparse_moe.routed_expert_*","*vision_tower*","*mm_projector*","layers.*.self_attn.fused_qkv_a_proj","layers.*.self_attn.q_b_proj","layers.*.self_attn.kv_b_proj","layers.*.self_attn.o_proj","layers.*.mlp.gate_up_proj","layers.*.mlp.down_proj","context_proj"]}' |
There was a problem hiding this comment.
does this acutally pass KVV https://github.com/MoonshotAI/Kimi-Vendor-Verifier
| PYTHONHASHSEED: '0' | ||
| AITER_LOG_LEVEL: WARNING | ||
| AITER_SITUV2_A4W4: '1' | ||
| AITER_QUICK_REDUCE_QUANTIZATION: INT4 |
There was a problem hiding this comment.
does this pass the entire suite of KVV (OCRBench, MMMU Pro Vision BEAM (1M), DeepSWE)? if so, plz send logs and manual reprod scripts
There was a problem hiding this comment.
We ran the full Kimi Vendor Verifier (KVV @ 66092cf) against this exact serving config on one MI355X node (TP8). We used the image this PR pins (rocm/atom-dev:nightly_202610310756@sha256:1c20b688…) with the same Quark online quantization (online_quant_config, global ptpc_fp8, same exclusions), INT4 quick all-reduce, FP8 KV, the same AITER/ATOM flags, and the DSpark draft. Client settings follow the KVV README for K3 in open-source mode: thinking on, thinking_effort: max, temperature 1.0, top_p 0.95.
| Benchmark | This config | KVV vendor table | Moonshot |
|---|---|---|---|
| OCRBench | 0.864 ± 0.011 | 0.878 – 0.897 | 0.89 |
| MMMU Pro Vision | 0.817 ± 0.009 | 0.804 – 0.820 | 0.82 |
| BEAM (1M) | 0.258 | 0.291 – 0.322 | 0.31 |
| DeepSWE | 0.646 (73 / 113) | 0.658 – 0.695 | 0.675 |
Quantization: the same techniques as the merged vLLM Kimi-K3 MI355X recipes
- INT4 all-reduce. Both vLLM recipes set INT4 quick all-reduce:
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: INT4in the single-node recipe and in the P/D recipe. This PR selects the same AITER quick-reduce mode withAITER_QUICK_REDUCE_QUANTIZATION: INT4. The sign-off check also lists Kimi-K3 as grandfathered for integer all-reduce. - Online FP8 quantization of BF16 weights.
- The vLLM single-node recipe quantizes online to FP8 per channel:
quantization-config: {"targets":{"*in_proj_qkvgfab":"fp8_per_channel"}}(L128). - This PR uses ATOM's Quark online quantization (
ptpc_fp8: FP8 per-channel weights, per-token activations) on the BF16 linears. - The exclusions keep the MXFP4 experts, the latent expert projections, the vision tower and the DSpark draft out of it.
- The vLLM single-node recipe quantizes online to FP8 per channel:
- FP8 MoE stage-2 output and FP8 KV are set in both vLLM recipes as well:
AITER_FLYDSL_STAGE2_FP8: '1'(single-node L56, P/D L83).
They do not move accuracy. We used the same image and client and changed only the server switches:
| Server settings | OCRBench |
|---|---|
This PR (Quark ptpc_fp8 online quantization + INT4 all-reduce) |
0.864 ± 0.011 |
Without ptpc_fp8 |
0.868 ± 0.011 |
| Every precision switch off (no online quantization, BF16 all-reduce, BF16 expert activations, BF16 KV, fp32 KDA state, BF16 prefill attention) | 0.849 ± 0.011 |
The three runs are within noise of each other.
- MMMU Pro Vision is within the vendor range.
- DeepSWE:
- 73 of 113 tasks were solved. One task is worth 0.009 and a single run's standard error is about 0.045, so this is within run-to-run noise of the vendor rows.
- Harness: Pier 0.3.1 with kimi-code 0.23.6, the version KVV names. There were no infrastructure failures.
- Our hosts firewall ports 80/443, so we let Pier's egress proxy reach the model server on its port. That is the only harness change, and the patch is included.
- BEAM:
- With KVV's default budget, 16 of the 35 conversations render past the 1,048,576-token window. K3's chat framing adds 25–40k tokens beyond the script's 5k allowance.
- We ran those 16 with KVV's own
--max-context-tokens 1000000, which drops the oldest 4–5% of their messages. The 19 conversations that fit at the defaults score 0.277. - KVV does not name the judge model behind its table. We used openai/gpt-oss-120b (reasoning effort high), so absolute scores may not compare directly.
Logs and manual repro scripts: https://github.com/Jasen2201/InferenceX/releases/tag/kimik3-atom-kvv-20261101
- the server launch script, with the exact flags and the A/B switches;
- the KVV client runner, one command per benchmark;
- the Pier kimi-code adapter and its patch;
- per-sample OCRBench and MMMU results;
- all 700 BEAM answers with judge scores;
- DeepSWE per-task results with full agent transcripts.
On the tool-call eval that blocked this PR: the fix is upstream in ROCm/ATOM#2849, and this PR now pins the nightly that includes it. The partial sweep 41183113911 shows kimi_tool_call_schema_full at 0.978–0.988 on all four 1P1D points, up from 0.934 at c1. Could you re-apply the sweep label so the remaining 1P2D points can finish?
There was a problem hiding this comment.
@Jasen2201 hmm what is the reason BEAM is so low compared to sglang/vllm official?
0.258 vs vLLM official 0.3188 vs SGLang official 0.3107
Is the eval setup the same? can yall chat with moonshot and/or inferact (official AMD partner) on what their official setup is verus ur setup?
There was a problem hiding this comment.
we would appreipcate it if u can provide this comparsion for BEAM & deepswe too plz?
There was a problem hiding this comment.
Both vLLM recipes set INT4 quick all-reduce
Yes @Jasen2201 , ever since we started the workstream (based on feedback from upstream community & from folks that deploy inference in production) to set an quality bar that Kimi (KVV) set out and the quality bar that AMD customers set, we have been asking the same questions on AMD vLLM & NVIDIA vLLM Kimi K3 PRs also
…tom-disagg-agentx # Conflicts: # inferencex-e2e/perf-changelog.yaml
Jasen2201
added
the
full-sweep-enabled
|
we will also run comparative experiments between Atom and Vllm&SGLang. These experiments are GPU-intensive and time-consuming; we will share the results later. Our goal is to ensure ATOM achieves precision parity with vLLM or SGLang. @functionstackx |
Thanks @Jasen2201 for running the KVV, appreipcate that you are supportive of the recent workstream to improve InferenceX quality bar. I am glad you took the first step in running the entire KVV eval suite. The goal should be to match the quality bar set out by the official KVV suite. Based on your comments, we chatted with a couple of inference experts in the community that have passed the entire KVV eval suite and they say that your reported ATOM's 0.258 on BEAM (1M) is significant below the quality bar for Kimi K3 BEAM KVV and suggest providing an break down of running it without any of these non-original quantization. We are happy to forward any reasonable questions to the inference experts in the community that have passed the entire KVV eval suite if you have anything else. Looking forward to seeing it improve in quality i sent a couple additional thoughts in slack that you may find useful (reach out to @chunfangamd ) |
functionstackx
removed
the
full-sweep-enabled
@Jasen2201 Agree , developing & maintaining an engine that matches the quality bar (Moonshot's KVV) of production customers is GPU-intensive and time-consuming |
Summary
Adds
kimik3-fp4-mi355x-atom-disagg-agentic: six Kimi-K3 MXFP4 AgentX points on MI355X with ATOM prefill/decode disaggregation behind AToMesh and Mooncake RDMA from prefill to decode. The existing ATOM entry,kimik3-fp4-mi355x-atom-agentic-mtp, serves prefill and decode on one node; these points split them.lmcache_offloadbeside Mooncake, so the paged KV and the KDA state images share one LMCache CPU pool. On the DSpark4 points,OFFLOAD_STATE_CPU_SIZE=128caps the state images within the 224 GB pool andATOM_STATE_CHECKPOINT_DEMAND=1adds checkpoints at observed shared prefixes; the fixed-interval grid stays off.dram-utilization0.598 and 0.343 budget 1794 GB and 1029 GB for the eight prefill ranks. Decode runs no CPU tier. ReplaySSM is on in both roles at the DCP8 points, andAITER_REUSE_IDENTICAL_COMM_GROUPS=1at C72 and C88.LMCACHE_LOCAL_CPU_USE_HUGEPAGES). The recipe'shost_setupreserves them on the prefill node throughhugepage_sanity.sh(4 ranks x the pool size per NUMA node, with 400 GiB left free for model loading), andteardownreleases them. A 4 KiB pool registered withhipHostRegisterstays movable, and each page the kernel migrates stalls the rank's GPU queues; that is what left LMCache prefills stuck in their startup barrier onmi355x-amds.Inferact/Kimi-K3-DSparkis served as it ships, in BF16: the target checkpoint's compressed-tensors ignore list (self_attn,mlp.*_proj) and theonline_quant_configexclusions cover every draft linear (layers.*andcontext_proj), as in the single-node ATOM recipe. Like that recipe, the draft caches in the target's FP8 MLA KV pool. Details with source links are in the draft data-type comment. The draft is read from/it-share/data/Inferact-Kimi-K3-DSpark, the copy the vLLM P/D recipe mounts. The srt driver supplies the golden AL.policy: random, so a 1P2D request picks a decode at random.rocm/atom-dev:nightly_202610310756, pinned by digest (ROCm/ATOM e281d58ac). It adds fix(kimi-k3): hold tool_choice="required" to the tools channel on the GPU ROCm/ATOM#2849, which forces Kimi-K3's tools channel on the GPU fortool_choice="required", the mode the Kimi tool-call eval sends. The previous image,nightly_202610190554(ROCm/ATOM daa4f1bb8), predates it and scored 0.9338 on the c1 tool-call eval.routed_expert_down_projandrouted_expert_up_projare stored in BF16 and run in BF16 on this image; daa4f1bb8 quantized them to MXFP4 at load.Validation. Full sweep 41183113911 on head 1a26087 (image
nightly_202610310756) is running. The earlier sweep 40782441830 ran onnightly_202610190554; it passed all six points, the canary and the evals, with c1 scoring 0.9338 on the tool-call eval./reuse-sweep-run 40782441830predates the image change, so merging needs/use 41183113911once the new sweep is green.Risks.
mi355x-amds(3 of 8 starts). These prefills pin up to 1.75 TiB; the hugepage-backed pools above are the fix, and the sweep above started every prefill.Reference: ATOM CI. The same topologies, server flags and golden ALs on ROCm/ATOM's MI355X CI (a different cluster), with one difference: the CI quantizes the DSpark draft to FP8, which this recipe does not. Expect these points to land somewhat lower. Total tok/s per GPU (input incl. cached, plus output), p90 TPOT:
AI model disclosure
claude-opus-5-5(Claude Opus 5.5), running in Claude Code.infx.srt_slurm.workload, loaded it with srtctl and checked the rendered ATOM command lines and golden AL against the CI runs; checked draft-layer coverage against the CI's online-quantization records.claude-opus-5-5(Claude Opus 5.5), running in Claude Code (changelog sync, 2026-10-29).claude-opus-5-5(Claude Opus 5.5), running in Claude Code (image update, 2026-10-31).nightly_202610310756by digest, merged main, recorded the BF16 latent MoE projections in the changelog and this description, ran the new sweep and re-ran its failed jobs.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概要
新增
kimik3-fp4-mi355x-atom-disagg-agentic：MI355X 上 6 个 Kimi-K3 MXFP4 AgentX 测试点，采用 ATOM 预填充/解码分离，经 AToMesh 路由，Mooncake RDMA 负责预填充到解码的传输。现有的 ATOM 条目kimik3-fp4-mi355x-atom-agentic-mtp在单节点上同时做预填充和解码；这些测试点把两者拆开。lmcache_offload，分页 KV 和 KDA 状态镜像共用一个 LMCache CPU 池。DSpark4 测试点用OFFLOAD_STATE_CPU_SIZE=128把状态镜像限制在 224 GB 池内的 128 GB，并用ATOM_STATE_CHECKPOINT_DEMAND=1在观察到的共享前缀处补充 checkpoint；固定间隔网格保持关闭。dram-utilization取 0.598 和 0.343，为 8 个预填充 rank 分别预算 1794 GB 和 1029 GB。解码不使用 CPU 层。DCP8 测试点在两个角色上都开启 ReplaySSM，C72 和 C88 设置AITER_REUSE_IDENTICAL_COMM_GROUPS=1。LMCACHE_LOCAL_CPU_USE_HUGEPAGES）。配方的host_setup通过hugepage_sanity.sh在预填充节点上预留这些页（每个 NUMA 节点 4 个 rank x 池大小，并为模型加载保留 400 GiB 空闲内存），teardown负责释放。用hipHostRegister注册的 4 KiB 池仍可被迁移，内核每迁移一页都会让该 rank 的 GPU 队列停顿；这正是mi355x-amds上 LMCache 预填充卡在启动 barrier 的原因。Inferact/Kimi-K3-DSpark按发布时的 BF16 精度提供服务：目标 checkpoint 的 compressed-tensors ignore 列表（self_attn、mlp.*_proj）和online_quant_config的排除列表覆盖全部草稿线性层（layers.*和context_proj），与单节点 ATOM 配方一致。与该配方相同，草稿的 KV 写入目标的 FP8 MLA KV 池。带源码链接的细节见草稿数据类型评论。草稿从/it-share/data/Inferact-Kimi-K3-DSpark读取，即 vLLM P/D 配方挂载的同一份拷贝。golden AL 由 srt 驱动提供。policy: random，因此 1P2D 的请求随机选择一个解码节点。rocm/atom-dev:nightly_202610310756，按 digest 固定（ROCm/ATOM e281d58ac）。它包含 fix(kimi-k3): hold tool_choice="required" to the tools channel on the GPU ROCm/ATOM#2849：对tool_choice="required"的请求，在 GPU 上强制进入 Kimi-K3 的 tools 通道，Kimi 工具调用评测发送的正是这种请求。上一版镜像nightly_202610190554（ROCm/ATOM daa4f1bb8）早于该修复，其 c1 工具调用评测得分为 0.9338。routed_expert_down_proj和routed_expert_up_proj在 checkpoint 中以 BF16 存储，在本镜像上按 BF16 运行；daa4f1bb8 会在加载时把它们量化为 MXFP4。验证。 完整扫描 41183113911（head 为 1a26087，镜像
nightly_202610310756）正在运行。之前的扫描 40782441830 使用nightly_202610190554，6 个测试点、canary 和评测均通过，其中 c1 工具调用评测为 0.9338。/reuse-sweep-run 40782441830早于镜像变更，因此新扫描全部通过后，合入需要/use 41183113911。风险。
mi355x-amds上发现，LMCache 预填充注册 1 TiB pinned 池后可能在启动 barrier 卡住（8 次启动中 3 次）。这些预填充最多 pin 1.75 TiB；上述大页池即为修复，上面的扫描中每个预填充都正常启动。参考：ATOM CI。 在 ROCm/ATOM 的 MI355X CI（另一套集群）上运行相同拓扑、服务参数和 golden AL，唯一区别是 CI 把 DSpark 草稿量化为 FP8，而本配方不量化。因此这些测试点的结果预计会略低。吞吐为每 GPU 总 token/s（输入含命中缓存的部分，加输出），延迟为 p90 TPOT；数据和运行链接见上表。
AI 模型使用说明
claude-opus-5-5（Claude Opus 5.5），运行于 Claude Code。infx.srt_slurm.workload绑定每个变体并用 srtctl 加载，核对生成的 ATOM 命令行和 golden AL 与 CI 运行一致；依据 CI 的在线量化记录核对草稿层的排除覆盖。claude-opus-5-5（Claude Opus 5.5），运行于 Claude Code（changelog 同步，2026-10-29）。claude-opus-5-5（Claude Opus 5.5），运行于 Claude Code（镜像更新，2026-10-31）。nightly_202610310756，合并 main，在 changelog 和本描述中记录 BF16 潜空间 MoE 投影，运行新的扫描并重跑失败的任务。