Skip to content

perf(amd): DSv4.1-Flash ATOM AgentX on the 1110 nightly with MXFP8 KV, tuned delayer TP points, DTP2 at c64 and DEP2 at c128/c256 / DSv4.1-Flash ATOM AgentX 更新至 1110 nightly,MXFP8 KV,TP 点调 delayer,c64 DTP2,c128/c256 DEP2 - #3965

Merged
Oseltamivir merged 38 commits into
mainfrom
perf/dsv41flash-atom-1013-dpa
Nov 13, 2026

Conversation

@yhl-amd

@yhl-amd yhl-amd commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Moves dsv41flash-fp4-mi355x-atom-agentic-dspark to the 2026-11-10 ATOM nightly on an MXFP8 KV cache, tunes the prefill delayer on the plain-TP points through concurrency 32, and runs DTP2 at 64 and DEP2 at 128/256.

  • Image: rocm/atom-dev:nightly_202610121516 → rocm/atom-dev:nightly_202611101510 (ATOM e5b74bc86, ROCm 7.2.4, digest sha256:160a4ec49379e7f1980a4e783254366b1d5c643b012abca57a21c0bcbbfe83a7), in the master config and the AToMesh frontend image.
  • KV cache: BF16 → MXFP8 (kv_cache_dtype: fp8, feat(moe): route before the rows move; per-pass real row counts; V4.1 FP8 KV ROCm/ATOM#2725). The index cache stays FP4.
  • Plain TP points: TP2 at concurrency 1, 2, 16, 32 and TP4 at 1, 2, 4, 8, 32.
    • c1: ATOM_ENABLE_PREFILL_DELAYER=1, ATOM_PREFILL_DECODE_INTERVAL=1000000, max-num-seqs 2, decode graphs at [1, 2] (the C1 recipe of ATOM's recipes/Kimi-K3.md).
    • c2 to c32 turn on the prefill delayer with ATOM_PREFILL_DECODE_INTERVAL 192 at c2–c8, 96 at c16, 40 at TP2 c32 and 16 at TP4 c32 (the interval TP4 c32 already used).
  • DP attention points: DTP2 (DP attention, MoE over TP, no TBO) at 64; DEP2 (plus expert parallelism) at 128 and 256. They set GPU_MAX_HW_QUEUES=5, ATOM_DP_SESSION_AFFINITY=1 (AIPerf sends its session id), --dp-load-balance least_tokens with ATOM_DP_LB_REQ_EQUIV=512, ATOM_ENABLE_PREFILL_DELAYER=1 and ATOM_PREFILL_DECODE_INTERVAL=10. The c256 point also captures a 256 decode graph.
  • Server: timeout-keep-alive: 900, past uvicorn's 5 s default that reset AIPerf's pooled connections. Every point except c1 runs ATOM's default max-num-seqs.
  • Removed: TP2 c4, c6, c8 and c64.
  • Draft data types: no change. ATOM loads the DSpark draft (mtp.0-2 in the checkpoint) as shipped at both images: FP8 E4M3 attention, shared experts and main_proj with 32×32 UE8M0 scales, MXFP4 routed experts, BF16 router, norms and heads. The draft shares the target's KV pool, so its KV cache follows the target from BF16 to MXFP8.
  • Accuracy-sensitive optimizations: none used (reduced-precision collectives, online quantization, shared-expert quantization, non-native indexer sharing, non-native FP4 KV, sub-FP32 state cache). Evidence: perf(amd): DSv4.1-Flash ATOM AgentX on the 1110 nightly with MXFP8 KV, tuned delayer TP points, DTP2 at c64 and DEP2 at c128/c256 / DSv4.1-Flash ATOM AgentX 更新至 1110 nightly,MXFP8 KV,TP 点调 delayer,c64 DTP2,c128/c256 DEP2 #3965 (comment)

12 points in total.

Validation: run 43180092301 on 3906935f0: all 12 benchmark points and all 6 gsm8k evals pass (em_strict 0.971–0.976).

中文

把 dsv41flash-fp4-mi355x-atom-agentic-dspark 换成 2026-11-10 的 ATOM nightly,KV cache 改为 MXFP8,纯 TP 点到 c32 调 prefill delayer,c64 跑 DTP2,c128/c256 跑 DEP2。

共 12 个点。

验证:run 43180092301(3906935f0):12 个 benchmark 点和 6 个 gsm8k eval 全部通过(em_strict 0.971–0.976)。

AI model disclosure
  • Claude Opus 5.5 (claude-opus-5-5): made the config, recipe and changelog changes and prepared this PR.
  • Claude Opus 5.5 (claude-opus-5-5): rewrote the perf-changelog block against the diff and checked the draft data types and accuracy-sensitive optimizations in the pinned ATOM source (for seungrokj).

🤖 Generated with Claude Code

…ightly

Move dsv41flash-fp4-mi355x-atom-agentic-dspark to
rocm/atom-dev:nightly_202610131505 (ATOM 661ef7661, with ROCm/ATOM#2376
V4.1 DPA and prefill TBO). Run TP2 c32/64/128/256 and TP4 c32 with DP
attention, prefill TBO, DP session affinity and the prefill delayer; drop
TP2 c6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yhl-amd yhl-amd added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Oct 13, 2026
Comment thread inferencex-e2e/configs/amd-master.yaml Outdated
Comment on lines +1535 to +1536
- { tp: 2, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 4, 8, 16, 128, 256], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml }
- { tp: 2, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: draft_model, conc-list: [32, 64, 128, 256], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Operators lose the ability to select the new tp2 c128/c256 DP-attention points individually via test-config --exp-names, which now errors out for them. The new dp-attn:true row (amd-master.yaml:1536, conc-list includes 128, 256) overlaps the plain-TP row (:1535, now also listing 128, 256) at tp=2. Single-node exp-name generation in infx/matrix/generate.py:1196-1200 builds the exp-name without dp-attn, so both rows produce the identical exp-name dsv41flash_tp2_conc128_kvnone_spec-draft_model. filter_exp_names (generate.py:1572) then raises "Experiment name(s) matched multiple rows" for that name, unlike multinode's multinode_agentic_exp_name which appends a dpa tag. Fix: make single-node agentic exp-name include dp-attn (e.g. append a dpa suffix when Fields.DP_ATTN is true), covering every tp/conc pair where a plain and DP-attention row can overlap.

Why this was flagged

Trigger: running python -m infx.matrix.generate test-config --config-keys dsv41flash-fp4-mi355x-atom-agentic-dspark --exp-names dsv41flash_tp2_conc128_kvnone_spec-draft_model (the documented per-point validation workflow in .github/workflows/README.md) after this PR. generate.py:1196-1200 computes identical exp-name strings for the tp2/conc128 plain-TP row (amd-master.yaml:1535) and the tp2/conc128 DP-attention row (amd-master.yaml:1536), because dp-attn is not part of the formula. filter_exp_names (generate.py:1572) counts 2 matches and raises ValueError('Experiment name(s) matched multiple rows'), so the new c128/c256 points this PR adds cannot be exercised individually. On base, this entry had no tp/conc pair shared between dp-attn true/false rows, so exp-names there were unique; actual per-job result files stay distinct (benchmark-tmpl.yml:254 embeds dpa in RESULT_FILENAME_BASE), so only exp-name-based selection/identification breaks, not result storage.

Verification: The single-node agentic exp-name formula at inferencex-e2e/infx/matrix/generate.py:1196-1200 encodes only tp, conc, kv-offload and spec, not dp-attn. amd-master.yaml:1535 (dp-attn: false) and :1536 (dp-attn: true) both have tp=2, kv-offloading: none, spec: draft_model, and overlap at conc 128 and 256, producing the identical exp-name dsv41flash_tp2_conc128_kvnone_spec-draft_model (and conc256).

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

yhl-amd and others added 2 commits October 16, 2026 01:06
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml
#	inferencex-e2e/configs/amd-master.yaml
#	inferencex-e2e/perf-changelog.yaml
@yhl-amd yhl-amd changed the title perf(amd): DSv4.1-Flash ATOM AgentX DPA + prefill TBO from c32 on the 1013 nightly / DSv4.1-Flash ATOM AgentX 更新至 1013 nightly 并从 c32 起开启 DPA + prefill TBO perf(amd): DSv4.1-Flash ATOM AgentX DPA + prefill TBO from c32 on the 1015 nightly / DSv4.1-Flash ATOM AgentX 更新至 1015 nightly 并从 c32 起开启 DPA + prefill TBO Oct 15, 2026
yhl-amd and others added 2 commits October 22, 2026 23:23
…1013-dpa

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
rocm/atom-dev:nightly_202610221638 (sha256:32c81e56e468...; ATOM
0.1.9.dev11+gce46f6877, ROCm 7.2.4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yhl-amd
yhl-amd requested a review from zijiecode as a code owner October 22, 2026 16:48
@yhl-amd yhl-amd changed the title perf(amd): DSv4.1-Flash ATOM AgentX DPA + prefill TBO from c32 on the 1015 nightly / DSv4.1-Flash ATOM AgentX 更新至 1015 nightly 并从 c32 起开启 DPA + prefill TBO perf(amd): DSv4.1-Flash ATOM AgentX DPA + prefill TBO from c32 on the 1022 nightly / DSv4.1-Flash ATOM AgentX 更新至 1022 nightly 并从 c32 起开启 DPA + prefill TBO Oct 22, 2026
yhl-amd and others added 9 commits October 24, 2026 00:16
… without TBO

Merge origin/main, move the image to rocm/atom-dev:nightly_202610231458
(sha256:05322b86...) and drop --enable-tbo from the DP-attention points:
TBO does not support SWA bounded replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the ATOM 2c772a5 release image (ROCm/ATOM run 39986380362,
sha256:01a19e8b...) instead of nightly_202610231458.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-num-seqs 128 queued half of the 256 sessions (TTFT 490 s); raise it to
256 and capture a 256 decode graph on that point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…seqs as B300 vLLM

Move to the ATOM 11a2d56 release image (ROCm/ATOM run 40072622970,
sha256:42716923...) and set max-num-seqs per point (per DP rank) like the
B300 vLLM AgentX recipe: 1 at c1, 2x concurrency for c2-c16, concurrency
from c32 up; capture sizes stop at each point's max-num-seqs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1013-dpa

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
max-num-seqs is twice the concurrency at c1 too (TP2 and TP4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop every max-num-seqs setting; decode graphs go back to the base list
(sizes through 32 plus 48, 64, 128), with 256 added on the c256 points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 42858547738

@seungrokj

Copy link
Copy Markdown
Collaborator

/use 42858547738

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

@seungrokj staged run 42858547738.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

seungrokj and others added 3 commits November 11, 2026 14:43
…e diff

- Update the image from rocm/atom-dev:nightly_202610121516 to nightly_202611101510 (ATOM e5b74bc86, ROCm 7.2.4) in the master config, and the AToMesh frontend container_image from sha256:a60811a8... to sha256:160a4ec4..., the same image.
- Replace plain TP2 at concurrency [1, 2, 4, 6, 8, 16, 32, 64] and plain TP4 at [1, 8, 32] with plain TP2 at [1, 2, 16, 32], DTP2 at 64, DEP2 at [128, 256] and plain TP4 at [1, 2, 4, 8]. TP2 c4, c6, c8 and c64 and TP4 c32 are dropped; TP4 c2 and c4 are new.
- Switch the KV cache from BF16 to MXFP8 at every point; the index cache stays FP4.
- Drop max-num-seqs 128. Concurrency 1 runs max-num-seqs 2, decode graphs [1, 2] and ATOM_PREFILL_DECODE_INTERVAL=1000000, the C1 recipe of ATOM's recipes/Kimi-K3.md (ROCm/ATOM#2659). The c256 point also captures 256.
- Set ATOM_PREFILL_DECODE_INTERVAL per point: 192 at TP2 c2 and TP4 c2-c8, 96 at TP2 c16, 40 at TP2 c32 and 10 at the DP points.
- The DP points add dp-load-balance least_tokens, ATOM_DP_SESSION_AFFINITY=1, ATOM_DP_LB_REQ_EQUIV=512 and GPU_MAX_HW_QUEUES=5, with AIPerf session ids as X-Dynamo-Session-Id.
- Every point adds timeout-keep-alive 900.
- The new ATOM brings V4.1 DP attention (#2376), the V4.1 MXFP8 KV cache with DSpark (#2725), DP prefill balancing and graph replay (#2551, #2696, #2697) and the default HIP Mega-MHC kernel (#2568).
- No change to the DSpark draft's weight precision; its KV cache follows the target from BF16 to MXFP8.

Also replace the recipe and master-config header comments with line comments (AGENTS.md "Comments"); no setting changes.

- 镜像从 rocm/atom-dev:nightly_202610121516 更新为 nightly_202611101510(ATOM e5b74bc86,ROCm 7.2.4),AToMesh 前端 container_image 同步更新为同一镜像。
- 纯 TP2 [1, 2, 4, 6, 8, 16, 32, 64] 和纯 TP4 [1, 8, 32] 改为纯 TP2 [1, 2, 16, 32]、DTP2 c64、DEP2 [128, 256] 和纯 TP4 [1, 2, 4, 8]。删除 TP2 c4、c6、c8、c64 和 TP4 c32;新增 TP4 c2 和 c4。
- 所有点的 KV cache 从 BF16 改为 MXFP8;index cache 仍为 FP4。
- 去掉 max-num-seqs 128。并发 1 使用 max-num-seqs 2、decode 图 [1, 2] 和 ATOM_PREFILL_DECODE_INTERVAL=1000000,即 ATOM recipes/Kimi-K3.md 的 C1 recipe(ROCm/ATOM#2659)。c256 额外捕获 256。
- 各点设置 ATOM_PREFILL_DECODE_INTERVAL:TP2 c2 和 TP4 c2-c8 为 192,TP2 c16 为 96,TP2 c32 为 40,DP 点为 10。
- DP 点增加 dp-load-balance least_tokens、ATOM_DP_SESSION_AFFINITY=1、ATOM_DP_LB_REQ_EQUIV=512 和 GPU_MAX_HW_QUEUES=5,AIPerf 以 X-Dynamo-Session-Id 发送 session id。
- 所有点增加 timeout-keep-alive 900。
- 新 ATOM 带来 V4.1 DP attention(#2376)、V4.1 MXFP8 KV cache 与 DSpark(#2725)、DP prefill 均衡与图回放(#2551、#2696、#2697)以及默认 HIP Mega-MHC kernel(#2568)。
- DSpark draft 权重精度不变;其 KV cache 随 target 从 BF16 变为 MXFP8。

另将 recipe 和 master config 的头部注释改为行注释(AGENTS.md "Comments"),不改变任何设置。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj

Copy link
Copy Markdown
Collaborator

Draft data types: no change. The DSpark draft (mtp.0–mtp.2 in deepseek-ai/DeepSeek-V4.1-Flash@314113f8, shards 44–46) is stored as follows (read from the safetensors headers):

Part Data type
Attention wq_a / wq_b / wkv / wo_a / wo_b, shared experts, main_proj FP8 E4M3 with 32×32 UE8M0 block scales
Routed experts (128 per stage) MXFP4 (packed E2M1, UE8M0 scale per 1×32)
Router gate, norms, Markov and confidence heads BF16
Router bias, attn_sink, mHC hc_* tensors FP32
Embeddings / LM head none in the draft; shared with the target (dspark.py#L225)

At ATOM e5b74bc86 (the image's revision), each draft stage is a DraftBlock built from the target's Block. Attention projections and main_proj are allocated with native_quant_config() (FP8, 32×32 blocks): attention.py#L243, #L277 (wo_a stays FP8, comment at L272–L276), dspark.py#L129. The MoE uses make_v4_quant_config with no online quantization (dspark.py#L170): routed experts run FP4 per 1×32 with MXFP8 activations (moe.py#L65). The model rejects any other checkpoint format (config.py#L47). The mono binder checks every draft tensor against these shipped dtypes and copies nothing (mono/weights.py#L42). The recipe sets no online_quant_config, quantization override or hf-overrides, so the draft runs as shipped. The base ATOM a0527c9db uses the same load path (dspark.py#L184, attention.py#L252).

Two things change for the draft between the two images. Both apply equally to the target, and both are permitted:

Accuracy-sensitive optimizations: none used. Checked against the accuracy-sensitive optimization policy (#4705). Every serving role was checked: plain TP2/TP4, DTP2 c64, DEP2 c128/c256 and the DSpark draft. Image defaults were read at ATOM e5b74bc86 and AITER 541185a7 (from the pip freeze in the run's server fingerprint):

Optimization Effective setting (base → head) Evidence
Reduced-precision all-reduce / reduce-scatter Off → off. AITER_QUICK_REDUCE_QUANTIZATION is unset, the default NONE disables quick reduce, and ATOM forces INT4 only for llama. Custom AR does not quantize. Comm-fused MoE is off (ATOM_MOE_GU_ITLV=0, EP excluded, no 5120-dim winner row). MoRI EP dispatch carries the native MXFP8 expert activations, and combine stays BF16. quick_all_reduce.py#L374, model_runner.py#L790, comm_fused_moe.py#L73, moe.py#L537-L545, moe.py#L688
Online linear-layer quantization Not used → not used. DeepSeek-V4.1 rejects online_quant_config, and the run logs online_quant_config: None. config.py#L398
Shared-expert quantization FP8 as shipped → FP8 as shipped. Folding into the MXFP4 routed GEMM is opt-in via ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4, which defaults to 0. quant_spec.py#L242-L259, envs.py#L1037-L1038
Non-native cross-indexer sharing Only the native sharing → only the native sharing (index_source_layer_ids, kv_source_layer_ids from config.json). ATOM's extra top-k skip needs use_index_cache, which is unset. deepseek_v4/__init__.py#L528-L543, config.py#L1033-L1041
Non-native FP4 KV cache Main KV BF16 → MXFP8, not FP4. The FP4 index cache is unchanged and native: the reference quantizes indexer K with fp4_act_quant(k, 32). reference model.py#L546, v41_pool_geometry.py#L176-L187
SSM/KDA/Mamba state cache below FP32 N/A: no recurrent state. The CSA compressor state stays FP32. v41_pool_geometry.py#L363-L373

No model-trainer veto was found on this PR or in linked issues. Not verified: the base image's AITER commit, and the server logs of the DTP2/DEP2 points (their collective settings come from code defaults and the recipe env).

中文

Draft 数据类型:无变化。 DSpark draft(deepseek-ai/DeepSeek-V4.1-Flash@314113f8 中的 mtp.0–mtp.2,shard 44–46)的存储类型如下(读自 safetensors header):

部分 数据类型
Attention wq_a / wq_b / wkv / wo_a / wo_b、shared experts、main_proj FP8 E4M3,32×32 UE8M0 block scale
Routed experts(每层 128 个) MXFP4(打包 E2M1,每 1×32 一个 UE8M0 scale)
Router gate、norm、Markov 和 confidence head BF16
Router bias、attn_sink、mHC hc_* FP32
Embedding / LM head draft 不含,与 target 共享(dspark.py#L225)

在 ATOM e5b74bc86(镜像版本)中,每个 draft 层是由 target Block 构造的 DraftBlock。attention 投影和 main_proj 用 native_quant_config()(FP8、32×32 block)分配:attention.py#L243、#L277(wo_a 保持 FP8)、dspark.py#L129。MoE 使用不带在线量化的 make_v4_quant_config(dspark.py#L170),routed experts 以 FP4(1×32)+ MXFP8 激活运行(moe.py#L65)。模型拒绝其他 checkpoint 格式(config.py#L47)。mono binder 按原始 dtype 校验每个 draft 张量,不做任何拷贝(mono/weights.py#L42)。recipe 未设 online_quant_config、量化覆盖或 hf-overrides,因此 draft 按原样运行。base ATOM a0527c9db 的加载路径相同(dspark.py#L184、attention.py#L252)。

两个镜像之间 draft 有两处变化,均同样作用于 target,且均被允许:

精度敏感优化:均未使用。 已对照精度敏感优化政策(#4705)检查所有 serving 角色:纯 TP2/TP4、DTP2 c64、DEP2 c128/c256 和 DSpark draft。镜像默认值读自 ATOM e5b74bc86 和 AITER 541185a7(取自该 run server fingerprint 中的 pip freeze):

优化 实际设置(base → head) 证据
低精度 all-reduce / reduce-scatter 关 → 关。AITER_QUICK_REDUCE_QUANTIZATION 未设,默认 NONE 即关闭 quick reduce,ATOM 只对 llama 强制 INT4。custom AR 不量化。comm-fused MoE 关闭(ATOM_MOE_GU_ITLV=0、EP 被排除、无 5120 维 winner 行)。MoRI EP dispatch 传输原生 MXFP8 专家激活,combine 保持 BF16。 quick_all_reduce.py#L374、model_runner.py#L790、comm_fused_moe.py#L73、moe.py#L537-L545、moe.py#L688
在线线性层量化 未用 → 未用。DeepSeek-V4.1 拒绝 online_quant_config,运行日志为 online_quant_config: None。 config.py#L398
Shared-expert 量化 原样 FP8 → 原样 FP8。融合进 MXFP4 routed GEMM 需 ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4,默认为 0。 quant_spec.py#L242-L259、envs.py#L1037-L1038
非原生跨 indexer 共享 仅原生共享 → 仅原生共享(config.json 中的 index_source_layer_ids、kv_source_layer_ids)。ATOM 额外的 top-k 跳过需要 use_index_cache,未设置。 deepseek_v4/__init__.py#L528-L543、config.py#L1033-L1041
非原生 FP4 KV cache 主 KV BF16 → MXFP8,不是 FP4。FP4 index cache 不变且为原生:参考实现用 fp4_act_quant(k, 32) 量化 indexer K。 参考 model.py#L546、v41_pool_geometry.py#L176-L187
SSM/KDA/Mamba 状态低于 FP32 不适用:无循环状态,CSA compressor 状态保持 FP32。 v41_pool_geometry.py#L363-L373

本 PR 及关联 issue 中未发现模型训练方否决。未验证:base 镜像的 AITER commit,以及 DTP2/DEP2 点的 server 日志(其通信设置依据代码默认值和 recipe env 判断)。

@seungrokj seungrokj left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Signed: seungrokj

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @seungrokj is a listed owner of inferencex-e2e/configs/amd-master.yaml and /inferencex-e2e/benchmarks/*_node/srt-slurm-recipes/**/mi355x-*/; perf-changelog.yaml falls only under the * catch-all.

✅ Check 1 (Sweep + evals on in-PR commit): PASS — run 43180092301 on in-PR commit 3906935f passed all 12 points (11 agentic / and the TP2 c1 canary /) and all 6 agentic eval / jobs. The only later commit, merge 8c6973b6, leaves this PR's recipe, master entry and changelog entry byte-identical. It brings in klaud, collectivex and other configs' changes, none of which reach this config's launch, benchmark or eval path.

✅ Check 2 (Evals pass): PASS — gsm8k ran on rocm/atom-dev:nightly_202611101510 with real acceptance. em_strict: TP2 c1 0.973, TP2 c16 0.972, TP2 c32 0.973, TP4 c8 0.971, TP4 c32 0.976, DEP2 c256 0.974. Every row is within 2σ of its best sibling and of the exact ingested cohort (0.973, run 41290411954). The MXFP8 KV cache is set in the base recipe, so every eval used it, and the DEP2 c256 eval covers DP attention + EP.

➖ Check 3 (Upstream recipe): N/A — the only affected config is framework: atom on MI355X, so no vLLM/SGLang production serving configuration is affected. The linked ROCm/ATOM#3204 is informational.

✅ Check 4 (Reuse command): PASS — seungrokj (permission and role write) posted /use 43180092301 on its own line. That is the Check 1 run.

✅ Check 5 (Latest checklist template): PASS — all 20 items of the current PR_REVIEW_CHECKLIST.md template are present and checked.

✅ Check 6 (Upstream images / main-branch / engine-first): PASS — (a) does not apply to ATOM. (b) open-source entries already exist for dsv41flash on cluster:mi355x-amds (dsv41flash-fp4-mi355x-vllm-agentic-dspark, dsv41flash-fp4-mi355x-sglang-agentic-dspark). (c) the image's ATOM commit e5b74bc86 is an ancestor of ROCm/ATOM main (41 commits behind).

✅ Check 7 (Deprecated models/scenarios): PASS — dsv41flash agentic coding is active in MODELS.md as of 2026-11-13.

✅ Check 8 (No architecture hacks): PASS — the recipe has no hf-overrides, online quantization, or indexer skipping/sharing knobs, and the FP4 index cache is native to the model. dsv41flash appears in the integer all-reduce baseline matrix at 2a45791d, so it is grandfathered. No quick-reduce quantization control is set.

✅ Check 9 (Spec-decode chat templates): PASS — agentx profile / AIPerf replays semianalysis_cc_traces_weka_062126 against the default chat endpoint with no raw-completions override (job 131093592172 log).

✅ Check 10 (No engine patches): PASS — the recipe has no patches, setup script, overlay, or wheel install, and no other reachable runtime path changes in this PR.

✅ Check 11 (Agentic golden AL): PASS — throughput runs apply roles.agg.args.spec-decode-acceptance-length=3.51 (job 131093592172 log). This equals the golden thinking_on K=5 value in deepseek-v4.1-flash__dspark__native__probabilistic-block.yaml. Evals keep real acceptance.

➖ Check 12 (Append-only): N/A — the new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — at ATOM e5b74bc86, the bundled DSpark draft (mtp.0-2) builds main_proj with native_quant_config(), and its MoE uses make_v4_quant_config with the unset online_quant_config. The draft therefore loads as shipped: FP8 attention, shared experts and main_proj; MXFP4 routed experts; BF16 router and heads. The draft's KV becomes MXFP8 only because it shares the target's KV pool, which the policy allows. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE does not apply to ATOM and is not set.

✅ Check 14 (Pareto coverage): PASS — one curve: dsv41flash / agentic-coding / mi355x_atom / fp4 / P90 E2EL, from run 43180092301 results_bmk at 3906935f, all 12 points on nightly_202611101510. The frontier has 7 of 12 measured points: TP4 c8, TP2 c16, TP4 c32, TP2 c32, DTP2 c64, DEP2 c128, DEP2 c256. This count comes from the raw artifact by hand; running the helper was not permitted in this environment.

✅ Check 15 (PR description): PASS — these claims in the body match the code and evidence: the image and digest, ATOM e5b74bc86, MXFP8 KV, per-point delayer intervals, DP settings, removed TP2 c4/c6/c8/c64, 12 points, and validation run 43180092301 on 3906935f0 (em_strict 0.971–0.976).

➖ Check 16 (NVIDIA overview): N/A — only MI355X (non-NVIDIA) running ATOM is affected.

➖ Check 17 (Metrics publication): N/A — ATOM does not use the Dynamo TRT-LLM metrics settings.

✅ Check 18 (Recipe, image, and sweep consistency): PASS — (a) every key generates at head. (b) the master TP2/TP4 plain, DTP2 c64 and DEP2 c128/c256 entries match the recipe overrides. (c) rocm/atom-dev:nightly_202611101510 resolves to sha256:160a4ec4…, the frontend's pinned digest. (d) and (e) do not apply. (f) the changelog matches the diff, including the image, MXFP8 KV, removed points and the unchanged draft precision. (g) none of the removed TP2 c4/c6/c8/c64 points is undominated: c4/c6/c8 lie under the TP2 c16–TP4 c8 segment, and c64 is dominated by DTP2 c64. (i) ATOM has no stream-interval knob in its arg_utils.py at e5b74bc86, so it streams every engine step, at most 6 tokens with 5 DSpark drafts. (h) report only, re-tuned deltas in tput/GPU and interactivity versus run 41290411954 on the old image: TP2 c1 +6.2%/+9.0%, TP2 c2 +4.9%/+10.5%, TP2 c16 +0.7%/+23.6%, TP2 c32 +1.0%/+26.4%, TP4 c1 +4.7%/+10.3%, TP4 c8 +2.6%/+16.2%, TP4 c32 +0.7%/+1.5%.

Assessed commit: 8c6973b6b530c153ee9a615a39c83353d1251a77.

@seungrokj

Copy link
Copy Markdown
Collaborator

recipe: ROCm/ATOM#3204

yhl-amd and others added 4 commits November 12, 2026 15:14
…val 40

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at interval 40"

This reverts commit cfc9ae2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…val 16

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yhl-amd yhl-amd added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 12, 2026
@seungrokj

Copy link
Copy Markdown
Collaborator

/use 43180092301

@seungrokj

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 43180092301

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

@seungrokj staged run 43180092301.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@Oseltamivir
Oseltamivir merged commit 8f29d8d into main Nov 13, 2026
35 checks passed
@Oseltamivir
Oseltamivir deleted the perf/dsv41flash-atom-1013-dpa branch November 13, 2026 03:16
@github-actions

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 12 0 12 0 0
eval 6 6 6 0 0
trace 12 0 0 0 0
power 12 0 0 0 0

benchmarkFiles 13, rawBenchmarkRows 24, coveredByPerJobFiles 12, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 143976, powerPublicationPoints 12, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants