Repository navigation
perf(amd): DSv4.1-Flash ATOM AgentX on the 1110 nightly with MXFP8 KV, tuned delayer TP points, DTP2 at c64 and DEP2 at c128/c256 / DSv4.1-Flash ATOM AgentX 更新至 1110 nightly,MXFP8 KV,TP 点调 delayer,c64 DTP2,c128/c256 DEP2 - #3965
Conversation
…ightly Move dsv41flash-fp4-mi355x-atom-agentic-dspark to rocm/atom-dev:nightly_202610131505 (ATOM 661ef7661, with ROCm/ATOM#2376 V4.1 DPA and prefill TBO). Run TP2 c32/64/128/256 and TP4 c32 with DP attention, prefill TBO, DP session affinity and the prefill delayer; drop TP2 c6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| - { tp: 2, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 4, 8, 16, 128, 256], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml } | ||
| - { tp: 2, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: draft_model, conc-list: [32, 64, 128, 256], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml } |
There was a problem hiding this comment.
🔴 Operators lose the ability to select the new tp2 c128/c256 DP-attention points individually via test-config --exp-names, which now errors out for them. The new dp-attn:true row (amd-master.yaml:1536, conc-list includes 128, 256) overlaps the plain-TP row (:1535, now also listing 128, 256) at tp=2. Single-node exp-name generation in infx/matrix/generate.py:1196-1200 builds the exp-name without dp-attn, so both rows produce the identical exp-name dsv41flash_tp2_conc128_kvnone_spec-draft_model. filter_exp_names (generate.py:1572) then raises "Experiment name(s) matched multiple rows" for that name, unlike multinode's multinode_agentic_exp_name which appends a dpa tag. Fix: make single-node agentic exp-name include dp-attn (e.g. append a dpa suffix when Fields.DP_ATTN is true), covering every tp/conc pair where a plain and DP-attention row can overlap.
Why this was flagged
Trigger: running python -m infx.matrix.generate test-config --config-keys dsv41flash-fp4-mi355x-atom-agentic-dspark --exp-names dsv41flash_tp2_conc128_kvnone_spec-draft_model (the documented per-point validation workflow in .github/workflows/README.md) after this PR. generate.py:1196-1200 computes identical exp-name strings for the tp2/conc128 plain-TP row (amd-master.yaml:1535) and the tp2/conc128 DP-attention row (amd-master.yaml:1536), because dp-attn is not part of the formula. filter_exp_names (generate.py:1572) counts 2 matches and raises ValueError('Experiment name(s) matched multiple rows'), so the new c128/c256 points this PR adds cannot be exercised individually. On base, this entry had no tp/conc pair shared between dp-attn true/false rows, so exp-names there were unique; actual per-job result files stay distinct (benchmark-tmpl.yml:254 embeds dpa in RESULT_FILENAME_BASE), so only exp-name-based selection/identification breaks, not result storage.
Verification: The single-node agentic exp-name formula at inferencex-e2e/infx/matrix/generate.py:1196-1200 encodes only tp, conc, kv-offload and spec, not dp-attn. amd-master.yaml:1535 (dp-attn: false) and :1536 (dp-attn: true) both have tp=2, kv-offloading: none, spec: draft_model, and overlap at conc 128 and 256, producing the identical exp-name dsv41flash_tp2_conc128_kvnone_spec-draft_model (and conc256).
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43180092301 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43180092301 |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml # inferencex-e2e/configs/amd-master.yaml # inferencex-e2e/perf-changelog.yaml
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
rocm/atom-dev:nightly_202610221638 (sha256:32c81e56e468...; ATOM 0.1.9.dev11+gce46f6877, ROCm 7.2.4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… without TBO Merge origin/main, move the image to rocm/atom-dev:nightly_202610231458 (sha256:05322b86...) and drop --enable-tbo from the DP-attention points: TBO does not support SWA bounded replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the ATOM 2c772a5 release image (ROCm/ATOM run 39986380362, sha256:01a19e8b...) instead of nightly_202610231458. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-num-seqs 128 queued half of the 256 sessions (TTFT 490 s); raise it to 256 and capture a 256 decode graph on that point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…seqs as B300 vLLM Move to the ATOM 11a2d56 release image (ROCm/ATOM run 40072622970, sha256:42716923...) and set max-num-seqs per point (per DP rank) like the B300 vLLM AgentX recipe: 1 at c1, 2x concurrency for c2-c16, concurrency from c32 up; capture sizes stop at each point's max-num-seqs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
max-num-seqs is twice the concurrency at c1 too (TP2 and TP4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop every max-num-seqs setting; decode graphs go back to the base list (sizes through 32 plus 48, 64, 128), with 256 added on the c256 points. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/reuse-sweep-run 42858547738 |
|
/use 42858547738 |
|
@seungrokj staged run 42858547738. This run remains available across future |
…e diff - Update the image from rocm/atom-dev:nightly_202610121516 to nightly_202611101510 (ATOM e5b74bc86, ROCm 7.2.4) in the master config, and the AToMesh frontend container_image from sha256:a60811a8... to sha256:160a4ec4..., the same image. - Replace plain TP2 at concurrency [1, 2, 4, 6, 8, 16, 32, 64] and plain TP4 at [1, 8, 32] with plain TP2 at [1, 2, 16, 32], DTP2 at 64, DEP2 at [128, 256] and plain TP4 at [1, 2, 4, 8]. TP2 c4, c6, c8 and c64 and TP4 c32 are dropped; TP4 c2 and c4 are new. - Switch the KV cache from BF16 to MXFP8 at every point; the index cache stays FP4. - Drop max-num-seqs 128. Concurrency 1 runs max-num-seqs 2, decode graphs [1, 2] and ATOM_PREFILL_DECODE_INTERVAL=1000000, the C1 recipe of ATOM's recipes/Kimi-K3.md (ROCm/ATOM#2659). The c256 point also captures 256. - Set ATOM_PREFILL_DECODE_INTERVAL per point: 192 at TP2 c2 and TP4 c2-c8, 96 at TP2 c16, 40 at TP2 c32 and 10 at the DP points. - The DP points add dp-load-balance least_tokens, ATOM_DP_SESSION_AFFINITY=1, ATOM_DP_LB_REQ_EQUIV=512 and GPU_MAX_HW_QUEUES=5, with AIPerf session ids as X-Dynamo-Session-Id. - Every point adds timeout-keep-alive 900. - The new ATOM brings V4.1 DP attention (#2376), the V4.1 MXFP8 KV cache with DSpark (#2725), DP prefill balancing and graph replay (#2551, #2696, #2697) and the default HIP Mega-MHC kernel (#2568). - No change to the DSpark draft's weight precision; its KV cache follows the target from BF16 to MXFP8. Also replace the recipe and master-config header comments with line comments (AGENTS.md "Comments"); no setting changes. - 镜像从 rocm/atom-dev:nightly_202610121516 更新为 nightly_202611101510(ATOM e5b74bc86,ROCm 7.2.4),AToMesh 前端 container_image 同步更新为同一镜像。 - 纯 TP2 [1, 2, 4, 6, 8, 16, 32, 64] 和纯 TP4 [1, 8, 32] 改为纯 TP2 [1, 2, 16, 32]、DTP2 c64、DEP2 [128, 256] 和纯 TP4 [1, 2, 4, 8]。删除 TP2 c4、c6、c8、c64 和 TP4 c32;新增 TP4 c2 和 c4。 - 所有点的 KV cache 从 BF16 改为 MXFP8;index cache 仍为 FP4。 - 去掉 max-num-seqs 128。并发 1 使用 max-num-seqs 2、decode 图 [1, 2] 和 ATOM_PREFILL_DECODE_INTERVAL=1000000,即 ATOM recipes/Kimi-K3.md 的 C1 recipe(ROCm/ATOM#2659)。c256 额外捕获 256。 - 各点设置 ATOM_PREFILL_DECODE_INTERVAL:TP2 c2 和 TP4 c2-c8 为 192,TP2 c16 为 96,TP2 c32 为 40,DP 点为 10。 - DP 点增加 dp-load-balance least_tokens、ATOM_DP_SESSION_AFFINITY=1、ATOM_DP_LB_REQ_EQUIV=512 和 GPU_MAX_HW_QUEUES=5,AIPerf 以 X-Dynamo-Session-Id 发送 session id。 - 所有点增加 timeout-keep-alive 900。 - 新 ATOM 带来 V4.1 DP attention(#2376)、V4.1 MXFP8 KV cache 与 DSpark(#2725)、DP prefill 均衡与图回放(#2551、#2696、#2697)以及默认 HIP Mega-MHC kernel(#2568)。 - DSpark draft 权重精度不变;其 KV cache 随 target 从 BF16 变为 MXFP8。 另将 recipe 和 master config 的头部注释改为行注释(AGENTS.md "Comments"),不改变任何设置。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Draft data types: no change. The DSpark draft (
At ATOM Two things change for the draft between the two images. Both apply equally to the target, and both are permitted:
Accuracy-sensitive optimizations: none used. Checked against the accuracy-sensitive optimization policy (#4705). Every serving role was checked: plain TP2/TP4, DTP2 c64, DEP2 c128/c256 and the DSpark draft. Image defaults were read at ATOM
No model-trainer veto was found on this PR or in linked issues. Not verified: the base image's AITER commit, and the server logs of the DTP2/DEP2 points (their collective settings come from code defaults and the recipe env). 中文Draft 数据类型:无变化。 DSpark draft(
在 ATOM 两个镜像之间 draft 有两处变化,均同样作用于 target,且均被允许:
精度敏感优化:均未使用。 已对照精度敏感优化政策(#4705)检查所有 serving 角色:纯 TP2/TP4、DTP2 c64、DEP2 c128/c256 和 DSpark draft。镜像默认值读自 ATOM
本 PR 及关联 issue 中未发现模型训练方否决。未验证:base 镜像的 AITER commit,以及 DTP2/DEP2 点的 server 日志(其通信设置依据代码默认值和 recipe env 判断)。 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Draft data types and accuracy-sensitive optimizations: #3965 (comment)
- Recipes:
Signed: seungrokj
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep + evals on in-PR commit): PASS — run 43180092301 on in-PR commit ✅ Check 2 (Evals pass): PASS — gsm8k ran on ➖ Check 3 (Upstream recipe): N/A — the only affected config is ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — all 20 items of the current ✅ Check 6 (Upstream images / main-branch / engine-first): PASS — (a) does not apply to ATOM. (b) open-source entries already exist for ✅ Check 7 (Deprecated models/scenarios): PASS — ✅ Check 8 (No architecture hacks): PASS — the recipe has no ✅ Check 9 (Spec-decode chat templates): PASS — ✅ Check 10 (No engine patches): PASS — the recipe has no patches, setup script, overlay, or wheel install, and no other reachable runtime path changes in this PR. ✅ Check 11 (Agentic golden AL): PASS — throughput runs apply ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — at ATOM ✅ Check 14 (Pareto coverage): PASS — one curve: dsv41flash / agentic-coding / mi355x_atom / fp4 / P90 E2EL, from run 43180092301 ✅ Check 15 (PR description): PASS — these claims in the body match the code and evidence: the image and digest, ATOM ➖ Check 16 (NVIDIA overview): N/A — only MI355X (non-NVIDIA) running ATOM is affected. ➖ Check 17 (Metrics publication): N/A — ATOM does not use the Dynamo TRT-LLM metrics settings. ✅ Check 18 (Recipe, image, and sweep consistency): PASS — (a) every key generates at head. (b) the master TP2/TP4 plain, DTP2 c64 and DEP2 c128/c256 entries match the recipe overrides. (c) Assessed commit: |
|
recipe: ROCm/ATOM#3204 |
…val 40 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at interval 40" This reverts commit cfc9ae2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…val 16 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/use 43180092301 |
|
/reuse-sweep-run 43180092301 |
|
@seungrokj staged run 43180092301. This run remains available across future |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 13, rawBenchmarkRows 24, coveredByPerJobFiles 12, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 143976, powerPublicationPoints 12, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Moves
dsv41flash-fp4-mi355x-atom-agentic-dsparkto the 2026-11-10 ATOM nightly on an MXFP8 KV cache, tunes the prefill delayer on the plain-TP points through concurrency 32, and runs DTP2 at 64 and DEP2 at 128/256.rocm/atom-dev:nightly_202610121516→rocm/atom-dev:nightly_202611101510(ATOMe5b74bc86, ROCm 7.2.4, digestsha256:160a4ec49379e7f1980a4e783254366b1d5c643b012abca57a21c0bcbbfe83a7), in the master config and the AToMesh frontend image.kv_cache_dtype: fp8, feat(moe): route before the rows move; per-pass real row counts; V4.1 FP8 KV ROCm/ATOM#2725). The index cache stays FP4.ATOM_ENABLE_PREFILL_DELAYER=1,ATOM_PREFILL_DECODE_INTERVAL=1000000,max-num-seqs 2, decode graphs at[1, 2](the C1 recipe of ATOM'srecipes/Kimi-K3.md).ATOM_PREFILL_DECODE_INTERVAL192 at c2–c8, 96 at c16, 40 at TP2 c32 and 16 at TP4 c32 (the interval TP4 c32 already used).GPU_MAX_HW_QUEUES=5,ATOM_DP_SESSION_AFFINITY=1(AIPerf sends its session id),--dp-load-balance least_tokenswithATOM_DP_LB_REQ_EQUIV=512,ATOM_ENABLE_PREFILL_DELAYER=1andATOM_PREFILL_DECODE_INTERVAL=10. The c256 point also captures a 256 decode graph.timeout-keep-alive: 900, past uvicorn's 5 s default that reset AIPerf's pooled connections. Every point except c1 runs ATOM's defaultmax-num-seqs.mtp.0-2in the checkpoint) as shipped at both images: FP8 E4M3 attention, shared experts andmain_projwith 32×32 UE8M0 scales, MXFP4 routed experts, BF16 router, norms and heads. The draft shares the target's KV pool, so its KV cache follows the target from BF16 to MXFP8.12 points in total.
Validation: run 43180092301 on
3906935f0: all 12 benchmark points and all 6 gsm8k evals pass (em_strict 0.971–0.976).中文
把
dsv41flash-fp4-mi355x-atom-agentic-dspark换成 2026-11-10 的 ATOM nightly,KV cache 改为 MXFP8,纯 TP 点到 c32 调 prefill delayer,c64 跑 DTP2,c128/c256 跑 DEP2。rocm/atom-dev:nightly_202610121516→rocm/atom-dev:nightly_202611101510(ATOMe5b74bc86,ROCm 7.2.4),master config 和 AToMesh 前端镜像都换。max-num-seqs 2、decode 图[1, 2](沿用 ATOMrecipes/Kimi-K3.md的 C1 recipe)。GPU_MAX_HW_QUEUES=5、ATOM_DP_SESSION_AFFINITY=1、least_tokens+ATOM_DP_LB_REQ_EQUIV=512、delayer interval 10;c256 额外捕获 256 的 decode 图。timeout-keep-alive: 900;除 c1 外都用 ATOM 默认max-num-seqs。mtp.0-2):attention、shared experts 和main_proj为 FP8 E4M3(32×32 UE8M0 scale),routed experts 为 MXFP4,router、norm 和 head 为 BF16。draft 与 target 共用 KV pool,其 KV cache 随 target 从 BF16 变为 MXFP8。共 12 个点。
验证:run 43180092301(
3906935f0):12 个 benchmark 点和 6 个 gsm8k eval 全部通过(em_strict 0.971–0.976)。AI model disclosure
claude-opus-5-5): made the config, recipe and changelog changes and prepared this PR.claude-opus-5-5): rewrote the perf-changelog block against the diff and checked the draft data types and accuracy-sensitive optimizations in the pinned ATOM source (for seungrokj).🤖 Generated with Claude Code