Skip to content

feat(agentx): add Qwen3.5 FP4 MI355X ATOM AgentX (TP4 + TP2) on AttnFP8-V2 and nightly_202611131030 - #4310

Open
zhuyuhua-v wants to merge 9 commits into
mainfrom
amd/qwen3.5-fp4-mi355x-atom-agentx-1018
Open

zhuyuhua-v wants to merge 9 commits into
mainfrom
amd/qwen3.5-fp4-mi355x-atom-agentx-1018

Conversation

@zhuyuhua-v

@zhuyuhua-v zhuyuhua-v commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Add qwen3.5-fp4-mi355x-atom-agentic-mtp: Qwen3.5-397B-A17B AgentX via ATOM on the amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 checkpoint (revision 410f915f) and rocm/atom-dev:nightly_202611131030 (ATOM 0.1.10.dev122+g9fec99786, AITER 43840aee7, built from ROCm/ATOM main; digest sha256:91c7d38b). The configuration follows the MXFP4 submission configuration in ROCm/ATOM recipes/Agentic-Qwen3.5.md (ROCm/ATOM PR 3255), plus INT4 quick all-reduce.

Points (MTP at 3 draft tokens, dram-utilization 1.0):

  • TP4: CONC 1, 4, 8, 16, 24, 32 and 40 on the GPU prefix cache; CONC 48, 56, 64 and 80 with the LMCache DRAM tier.
  • TP2: CONC 12 on the GPU prefix cache; CONC 16, 20, 24, 32 and 40 with the LMCache DRAM tier.
  • LMCache points: a 253 GiB pool per rank, of which the GDN state tier may keep up to 110 GiB, so the server holds 1,087 GB of host DRAM at TP4 and 543 GB at TP2, against shares of 1,499 GB and 749 GB.
  • From CONC 32, at both TP4 and TP2, the prefill delayer stall and TTFT caps are 15 ticks.

Policy-relevant settings:

  • INT4 quick all-reduce (AITER_QUICK_REDUCE_QUANTIZATION=INT4) on every point, as qwen3.5-fp8-mi355x-atom-agentic-mtp runs. Qwen3.5-397B-A17B is in the model support matrix at the all-reduce policy baseline (2a45791), so it is grandfathered under MODELS.md; the GSM8K floor (0.94) still applies.
  • The checkpoint is served as shipped, with no online_quant_config: attention (GDN in_proj/out_proj and full-attention q/k/v/o) in FP8, routed and shared experts in MXFP4; routers, lm_head and the whole MTP layer (mtp.*) stay BF16, so the draft runs in BF16 as shipped.
  • The image carries ROCm/aiter PR 6981 (tuned MXFP4 MoE rows for the TP4 expert shape) and ROCm/ATOM PRs 3106 and 3108.
中文

新增 qwen3.5-fp4-mi355x-atom-agentic-mtp:在 amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 权重(revision 410f915f)上用 ATOM 跑 Qwen3.5-397B-A17B AgentX,镜像 rocm/atom-dev:nightly_202611131030(ATOM 0.1.10.dev122+g9fec99786,AITER 43840aee7,ROCm/ATOM main 构建,digest sha256:91c7d38b)。配置沿用 ROCm/ATOM recipes/Agentic-Qwen3.5.md 中 MXFP4 提交配置(ROCm/ATOM PR 3255),另外打开 INT4 quick all-reduce。

测试点(MTP 3 个 draft token,dram-utilization 1.0):

  • TP4:CONC 1、4、8、16、24、32、40 只用 GPU prefix cache;CONC 48、56、64、80 加 LMCache DRAM 层。
  • TP2:CONC 12 只用 GPU prefix cache;CONC 16、20、24、32、40 加 LMCache DRAM 层。
  • LMCache 点:每 rank 253 GiB 池,其中最多 110 GiB 给 GDN state,TP4 占主机 DRAM 1,087 GB、TP2 占 543 GB,分别在 1,499 GB 和 749 GB 的份额之内。
  • CONC 32 起(TP4 与 TP2 均是),prefill delayer 的 stall 与 TTFT 上限为 15 ticks。

与政策相关的设置:

  • 所有点打开 INT4 quick all-reduce(AITER_QUICK_REDUCE_QUANTIZATION=INT4),与 qwen3.5-fp8-mi355x-atom-agentic-mtp 一致。Qwen3.5-397B-A17B 在 all-reduce 政策基线(2a45791d)的模型支持矩阵中,按 MODELS.md 属于豁免模型;GSM8K 门槛(0.94)仍然适用。
  • checkpoint 原样服务,不使用 online_quant_config:attention 为 FP8,路由与共享专家为 MXFP4,router、lm_head 与整个 MTP 层保持 BF16,draft 按发布时的 BF16 运行。
  • 镜像包含 ROCm/aiter PR 6981(TP4 专家形状的 MXFP4 MoE 调优行)以及 ROCm/ATOM PR 3106、3108。

Add qwen3.5-fp4-mi355x-atom-agentic-mtp: the Qwen3.5 FP8 ATOM AgentX
recipe (ROCm/ATOM PR 2592, InferenceX PR 4267) on the
amd/Qwen3.5-397B-A17B-MXFP4 checkpoint and
rocm/atom-dev:nightly_202610181506. TP4 with MTP at 3 draft tokens,
CONC 1-40 on the GPU prefix cache and CONC 48-80 with the LMCache DRAM
tier (253 GiB KV + 110 GiB GDN state per rank).

中文:新增 qwen3.5-fp4-mi355x-atom-agentic-mtp:将 Qwen3.5 FP8 ATOM
AgentX 配方(ROCm/ATOM PR 2592,InferenceX PR 4267)用于
amd/Qwen3.5-397B-A17B-MXFP4 权重,镜像 rocm/atom-dev:nightly_202610181506。
TP4,MTP 3 个 draft token,CONC 1-40 只用 GPU prefix cache,CONC 48-80
加 LMCache DRAM 层(每 rank 253 GiB KV + 110 GiB GDN state)。

Signed-off-by: zhuyuhua-v <yuhzhu@amd.com>
@zhuyuhua-v
zhuyuhua-v requested review from a team and a balanced review from Copilot October 19, 2026 09:42
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment on lines +146 to +149
LMCACHE_MAX_LOCAL_CPU_SIZE: '253'
LMCACHE_NUMA_MODE: auto
OFFLOAD_MIN_LOAD_TOKENS: '8192'
OFFLOAD_STATE_CPU_SIZE: '110'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The CONC 48-80 DRAM-offload points reserve about 1559 GB of host DRAM, more than the 1499 GB budget their results report. Each rank gets LMCACHE_MAX_LOCAL_CPU_SIZE '253' plus OFFLOAD_STATE_CPU_SIZE '110' (363 GiB, about 390 GB), times 4 ranks. Fix: keep KV plus GDN state at or below TOTAL_CPU_DRAM_GB / 4 per rank (374 GB, about 349 GiB), e.g. KV 239 GiB with state 110 GiB. Apply it to all 4 lmcache overrides (lines 146/149, 168/171, 190/193, 212/215). The recipe comment at line 54 already shows the overrun.

Why this was flagged

The matrix computes the TP4 share in generate.py:agentic_dram_offload_gb as MAX_AGENTIC_AVAILABLE_CPU_DRAM_MIB 2_861_022 x 1.0 x 4/8, which gives 1499 GB. The recipe pins TOTAL_CPU_DRAM_GB '1499', and single_node.py:138 checks that it matches. Each override at agentic.yaml:146-149, 168-171, 190-193 and 212-215 sizes 253 GiB of LMCache KV plus 110 GiB of GDN state per rank. Across 4 ranks that is 1452 GiB, or 1559 GB, which the comment at agentic.yaml:53-55 states itself. So the published CONC 48/56/64/80 results claim dram-utilization 1.0 with 1499 GB, but they use about 60 GB more DRAM than any comparable entry may. The srun step cap from gpu_share_mib (3_079_397 x 4/8 MiB, about 1614 GB) leaves only about 55 GB for 4 engine processes, the frontend and pinned buffers, so the job could also hit OOM. Nothing validates recipe-side buffer sizes against TOTAL_CPU_DRAM_GB. The sibling minimaxm3 recipe sizes per rank at TOTAL_CPU_DRAM_GB / TP.

Verification: Each lmcache override sets LMCACHE_MAX_LOCAL_CPU_SIZE '253' and OFFLOAD_STATE_CPU_SIZE '110' (lines 146/149, 168/171, 190/193, 212/215), so 4 TP ranks hold 1452 GiB, about 1559 GB. results/agentic/init.py:192 publishes allocated_cpu_dram_gb straight from TOTAL_CPU_DRAM_GB, so these results report 1499 GB. dram-utilization is already 1.0, so the metadata cannot be raised to match.

Comment thread inferencex-e2e/configs/amd-master.yaml Outdated
Comment on lines +426 to +429
# MI355X MXFP4 AgentX via ATOM with MTP (3 draft tokens), TP4 only: the FP8
# sibling's recipe (ROCm/ATOM PR 2592) on the MXFP4 checkpoint. CONC 1-40 serve
# off the GPU prefix cache; CONC 48-80 add the LMCache DRAM tier at 253 GiB KV +
# 110 GiB GDN state per rank.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Maintainers get new header and provenance comments, which the mandatory AGENTS.md "Code comments" rules forbid in configs and recipes. The master-entry header describes the config and its origin ("the FP8 sibling's recipe (ROCm/ATOM PR 2592)"). Fix: delete both header blocks and keep only short per-line technical reasons, for example on the LMCache sizing values. The history already lives in perf-changelog.yaml. Same pattern at 2 sites (amd-master.yaml:426-429, mi355x-fp4-mtp/agentic.yaml:51-55).

Why this was flagged

AGENTS.md:60-66 says these rules apply to every file, including recipes and configs. It bans header comments above a recipe, override or section, bans describing what the config or topology is, and bans provenance or history. amd-master.yaml:426-429 puts a 4-line header above qwen3.5-fp4-mi355x-atom-agentic-mtp. That header describes the topology (TP4 only, CONC ranges) and the provenance (FP8 sibling recipe, ROCm/ATOM PR 2592). agentic.yaml:51-55 puts a 5-line header above all overrides ("One variant per point, all TP4..."). It restates the cudagraph lists and env gating that the code already shows. The base branch has neither comment. After the merge, the repo carries comment types its mandatory rules prohibit, and they will go stale as values change.

Verification: Nit. It is a written-rule violation, so nothing breaks at runtime. AGENTS.md:60-65, "Code comments (mandatory)", forbids header comments and history. Site 1 is inferencex-e2e/configs/amd-master.yaml, a 4-line header that describes the config and records provenance. Site 2 is agentic.yaml, a 5-line header above override_tp4_c1:.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

All four DRAM variants configure approximately 1559 GB of host pools against a declared and reported 1499 GB budget.

1 open issue
What changed in this PR

Adds Qwen3.5 MXFP4 AgentX benchmarking on MI355X using ATOM’s October 18 image.

Changes:

  • Registers a TP4, three-draft-token MTP configuration.
  • Adds eleven concurrency points, with LMCache DRAM offloading at CONC 48–80.
  • Records the new benchmark coverage in the performance changelog.
中文

新增 MI355X 上的 Qwen3.5 MXFP4 ATOM AgentX 基准配置,使用 TP4 和三个 draft token 的 MTP,覆盖十一个并发点;CONC 48–80 启用 LMCache DRAM 卸载,并新增性能变更记录。

File Description
inferencex-e2e/​perf-changelog.yaml Records the new benchmark configuration.
inferencex-e2e/​configs/​amd-master.yaml Registers the model, image and search space.
inferencex-e2e/​benchmarks/​single_node/​srt-slurm-recipes/​qwen3.5/​atom/​mi355x-fp4-mtp/​agentic.yaml Defines serving settings and concurrency variants.

Review effort: Balanced

Comment on lines +146 to +149
LMCACHE_MAX_LOCAL_CPU_SIZE: '253'
LMCACHE_NUMA_MODE: auto
OFFLOAD_MIN_LOAD_TOKENS: '8192'
OFFLOAD_STATE_CPU_SIZE: '110'
@zhuyuhua-v zhuyuhua-v added AMD full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures agentx AgentX benchmarks, recipes, and infrastructure labels Oct 19, 2026
…x-atom-agentx-1018

Signed-off-by: zhuyuhua-v <yuhzhu@amd.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

…x-atom-agentx-1018

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@zhuyuhua-v
zhuyuhua-v requested a review from zijiecode as a code owner October 21, 2026 03:48
…x-atom-agentx-1018

# Conflicts:
#	inferencex-e2e/configs/amd-master.yaml
#	inferencex-e2e/perf-changelog.yaml
…int and nightly_202611071509

Refresh qwen3.5-fp4-mi355x-atom-agentic-mtp:

- Image rocm/atom-dev:nightly_202610181506 -> nightly_202611071509
  (ATOM 0.1.10.dev45+g6d22f7dbf, built from ROCm/ATOM main). The atomesh
  router container_image digest moves from 1bb1909a to 39ac8572, the same
  nightly as the workers.
- Checkpoint amd/Qwen3.5-397B-A17B-MXFP4 ->
  amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 (revision 410f915f), served as
  shipped with no online_quant_config. Attention (GDN projections and full
  attention q/k/v/o) is stored in PTPC FP8, routed and shared experts in
  MXFP4; routers, lm_head and the whole MTP layer stay BF16.
- Drop AITER_QUICK_REDUCE_QUANTIZATION=INT4, so all-reduce is no longer
  integer-quantized (pinned AITER defaults to NONE).

Points, TP4 shape, MTP 3 and LMCache 0.5.6.dev139+gf22dec28 are unchanged.

中文:刷新 qwen3.5-fp4-mi355x-atom-agentic-mtp:镜像换成
nightly_202611071509(ATOM main 构建),router digest 同步;模型换成
amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2,按 checkpoint 原样服务(attention
为 PTPC FP8,专家 MXFP4,router、lm_head 与 MTP 层保持 BF16);去掉
AITER_QUICK_REDUCE_QUANTIZATION=INT4,all-reduce 不再做整数量化。

Signed-off-by: zhuyuhua-v <yuhzhu@amd.com>
@zhuyuhua-v zhuyuhua-v changed the title feat(agentx): add Qwen3.5 FP4 MI355X ATOM AgentX on _1018 docker feat(agentx): add Qwen3.5 FP4 MI355X ATOM AgentX on AttnFP8-V2 and nightly_202611071509 Nov 8, 2026
@zhuyuhua-v

Copy link
Copy Markdown
Collaborator Author

/use 42460635358

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

@zhuyuhua-v staged run 42460635358.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…x-atom-agentx-1018

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@seungrokj

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 42460635358

@seungrokj

seungrokj commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Draft data types: no change. The MTP draft (mtp.* in amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 at 410f915f) is stored entirely in BF16 (read from the safetensors headers). The checkpoint's Quark config lists every mtp.* module in exclude:

Part Draft (mtp.*) Target, for comparison
fc BF16 n/a
Attention q/k/v/o BF16 full attention: FP8 E4M3, per-channel BF16 scales; GDN in_proj_*/out_proj: same
Routed experts BF16 MXFP4 (packed U8, UE8M0 scales per 32)
Shared expert gate/up/down BF16 MXFP4 (packed U8, UE8M0 scales per 32)
Router gate, shared_expert_gate, norms BF16 BF16
KV cache FP8 FP8

At ATOM 6d22f7dbf (the image's revision):

  • Qwen3_5MTP.__init__ re-indexes the checkpoint's mtp.layers.0.* excludes to the draft's absolute layer index (60).
  • get_layer_quant_config returns no-quant for excluded layers, so every draft linear and expert loads in BF16.
  • The draft KV pool uses the target's kv_cache_dtype (draft_kv.py), which is fp8 here.
  • The recipe sets no online_quant_config (the server log shows 'online_quant_config': None) and no draft dtype, so the draft runs as shipped.

DRAM note: the recipe comment's 1,559 GB total adds the 110 GiB state tier on top of the 253 GiB LMCache pool. At this commit the state tier is carved from that pool (state_tier_cpu_bytes, connector), and the log reads "sharing the paged-KV CPU pool". So the server holds 4 × 253 GiB = 1,087 GB, within the 1,499 GB TP4 share.

中文

Draft 数据类型:无变化。 MTP draft(amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 410f915f 中的 mtp.*)全部以 BF16 存储(读取自 safetensors header)。checkpoint 的 Quark 配置把所有 mtp.* 模块列入 exclude:

部分 Draft(mtp.*) Target(对照)
fc BF16 不适用
Attention q/k/v/o BF16 full attention:FP8 E4M3,per-channel BF16 scale;GDN in_proj_*/out_proj 相同
路由专家 BF16 MXFP4(U8 打包,每 32 个一组 UE8M0 scale)
shared expert gate/up/down BF16 MXFP4(U8 打包,每 32 个一组 UE8M0 scale)
router gate、shared_expert_gate、norm BF16 BF16
KV cache FP8 FP8

在 ATOM 6d22f7dbf(镜像对应版本)中:

  • Qwen3_5MTP.__init__ 把 checkpoint 的 mtp.layers.0.* 排除项重新映射到 draft 的绝对层号(60)。
  • get_layer_quant_config 对被排除的层返回不量化,因此 draft 的所有 linear 与专家都以 BF16 加载。
  • draft KV 池使用 target 的 kv_cache_dtype(draft_kv.py),此处为 fp8。
  • 配方未设置 online_quant_config(server log 显示 'online_quant_config': None),也未设置 draft dtype,因此 draft 按发布时运行。

DRAM 说明:配方注释中的 1,559 GB 把 110 GiB 的 state 层算在了 253 GiB 的 LMCache 池之外。在该版本中,state 层是从这个池中划出的(state_tier_cpu_bytes,connector),日志也显示 "sharing the paged-KV CPU pool"。因此 server 共占用 4 × 253 GiB = 1,087 GB,在 TP4 的 1,499 GB 份额之内。

@seungrokj seungrokj left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Signed: seungrokj

…ly_202611071509

- Add Qwen3.5-397B-A17B AgentX on MI355X via ATOM on rocm/atom-dev:nightly_202611071509 (ATOM 0.1.10.dev45+g6d22f7dbf, built from ROCm/ATOM main; the atomesh frontend container_image pins its digest sha256:39ac8572890358741585bf5281f7ce49117d99f9b8bc84a09b70749cf1df893e), serving amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 (revision 410f915f) as shipped: PTPC FP8 attention projections, MXFP4 routed and shared experts, BF16 routers, shared_expert_gate, lm_head and MTP layer.
- Eleven TP4 points with MTP at 3 draft tokens: CONC 1, 4, 8, 16, 24, 32 and 40 on the GPU prefix cache, and CONC 48, 56, 64 and 80 with LMCache 0.5.6.dev139+gf22dec28 DRAM offload at dram-utilization 1.0.
- Every point runs FP8 KV cache, prefix caching, max-num-seqs 1024, CUDA graphs for every batch size up to max(8, 2 x CONC), GDN state checkpoints every 16384 prompt tokens written mid-step (ATOM_GDN_STATE_MIDSTEP=1, ATOM_STATE_CHECKPOINT_END_MARGIN=5, ATOM_STATE_CHECKPOINT_DEMAND=0), FlyDSL paged decode, the QK-norm/RoPE/cache-quant fusion, and the prefill delayer at interval 4 with a 5000 ms queue cap; CONC 32 and up set the delayer stall and TTFT caps to 15 ticks.
- CONC 48-80 add the lmcache_offload connector with a 253 GiB LMCache CPU pool per rank (chunk 256, loads from 8192 tokens), of which the GDN state tier may keep up to 110 GiB; the state shares that pool, so the server holds 1,012 GiB (1,087 GB) of host DRAM against the 1,499 GB TP4 share.
- AITER_QUICK_REDUCE_QUANTIZATION is unset, so all-reduce stays unquantized (the pinned AITER f7b65674b defaults it to NONE and ATOM forces INT4 only for llama). The GDN conv and recurrent state are stored in BF16, the ATOM default for qwen3_5_moe_text, while the checkpoint specifies mamba_ssm_dtype float32. Throughput runs use the qwen3.5-397b-a17b-nvfp4__mtp__native__default-default golden AL (3.39).
- The recipe follows ROCm/ATOM recipes/Agentic-Qwen3.5.md (the FP8 checkpoint, ROCm/ATOM PR 2592) without its INT4 quick reduce, and needs ROCm/ATOM PRs 2549, 2588, 2589, 2590, 2591, 2599 and 2609, all in this image.
- The MTP draft runs as shipped: the checkpoint excludes every mtp.* layer from quantization and ATOM re-indexes those excludes to the draft's absolute layer numbers, so its attention, experts, shared expert and fc stay BF16; the draft KV cache is FP8 like the target's.
- Remove the header comments from the recipe and the master config (AGENTS.md comment rule).

- 新增 MI355X 上的 Qwen3.5-397B-A17B ATOM AgentX,镜像 rocm/atom-dev:nightly_202611071509(ATOM 0.1.10.dev45+g6d22f7dbf,ROCm/ATOM main;frontend digest sha256:39ac8572),原样服务 amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2(410f915f):attention 为 PTPC FP8,路由专家与 shared expert 为 MXFP4,router、lm_head 与 MTP 层为 BF16。
- 11 个 TP4 MTP3 点:CONC 1-40 只用 GPU prefix cache,CONC 48-80 在 dram-utilization 1.0 下加 LMCache DRAM offload。
- 所有点:FP8 KV、prefix caching、max-num-seqs 1024、CUDA graph 覆盖到 max(8, 2 x CONC)、每 16384 token 在 step 中途写 GDN state checkpoint、FlyDSL decode、QK-norm/RoPE/cache-quant 融合、prefill delayer(interval 4,队列上限 5000 ms);CONC 32 起 delayer 的 stall/TTFT 上限为 15 tick。
- CONC 48-80:每 rank 253 GiB LMCache 池,其中最多 110 GiB 给 GDN state 层,因此每台 server 占用 1,087 GB 主机 DRAM,在 TP4 的 1,499 GB 份额之内。
- 无整数 all-reduce(AITER f7b65674b 默认 NONE)。GDN state 为 BF16(ATOM 对 qwen3_5_moe_text 的默认值;checkpoint 的 mamba_ssm_dtype 为 float32)。Golden AL 3.39。
- 沿用 ROCm/ATOM Agentic-Qwen3.5.md(PR 2592),去掉 INT4 quick reduce;依赖 ATOM PR 2549、2588、2589、2590、2591、2599、2609,均已包含在该镜像中。
- MTP draft 按发布时的 BF16 运行(mtp.* 被排除在量化之外,并由 ATOM 重新映射层号);draft KV 为 FP8。
- 按 AGENTS.md 注释规则,删除 recipe 和 master config 中的头部注释。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Verdict: WARN ⚠️

⚠️ Check 15 (PR description): WARN — The body says CONC 48-80 use "253 GiB KV + 110 GiB GDN state per rank", but at ATOM 6d22f7dbf the 110 GiB state tier is carved from the 253 GiB LMCache pool it shares (state_tier_cpu_bytes), so each rank holds 253 GiB in total (4 x 253 GiB = 1,087 GB). The changelog entry at f4b5e7ee already says this; please correct the English and Chinese body text to match.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @seungrokj is a listed owner of inferencex-e2e/configs/amd-master.yaml and of the srt-slurm-recipes/**/mi355x-*/ recipe path; perf-changelog.yaml falls under the catch-all owner.

✅ Check 1 (Sweep on in-PR commit): PASS — In-PR commit 3fd52b00 has green, executed agentic / (all 11 points) and agentic eval / (c24, c40, c80) checks in run 42460635358 on nightly_202611071509. The later commits do not change runtime behavior: merge 2d823fe only brings in unrelated configs and result staging, and f4b5e7ee changes only recipe and master comments plus changelog wording.

✅ Check 2 (Evals): PASS — gsm8k em_strict is 0.9750 / 0.9773 / 0.9757 (c24 / c40 / c80) on the PR image. All rows are within 2 sigma of the best sibling and of the closest ingested qwen3.5 FP4 MTP cohort (run 42278371910, 0.9773). This is a new config with no conditional backend change.

➖ Check 3 (Upstream recipe): N/A — The only affected entry is framework: atom (qwen3.5-fp4-mi355x-atom-agentic-mtp); no vLLM or SGLang serving configuration is affected. The ROCm/ATOM recipe link is informational.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 42460635358 by seungrokj (COLLABORATOR) is the newest authorized command, after /use 42460635358 by zhuyuhua-v (COLLABORATOR). Both point to the Check 1 run.

✅ Check 5 (Checklist template): PASS — All 18 items and the sub-item of the current template are present and checked.

✅ Check 6 (Upstream images / ordering / provenance): PASS — (a) does not apply to ATOM. (b) The open-source entry qwen3.5-fp4-mi355x-sglang-agentic-mtp already exists for the same model and SKU. (c) The image's ATOM 6d22f7dbf is an ancestor of ROCm/ATOM main.

✅ Check 7 (Deprecated models): PASS — qwen3.5 agentic coding at fp4 is active in MODELS.md.

✅ Check 8 (No arch hacks): PASS — There are no hf_overrides, online_quant_config, or model edits. The FP8 KV cache and ATOM's default BF16 GDN state are covered by the passing c24/c40/c80 evals. qwen3.5 is grandfathered in the integer all-reduce baseline matrix, so (c) is exempt.

✅ Check 9 (Spec-decode chat template): PASS — The AgentX replay client runs with AIPERF_APPLY_CHAT_TEMPLATE: 'true' set in the recipe's benchmark env.

✅ Check 10 (No engine patches): PASS — No reachable patching. The recipe has no setup script, and the srt-slurm patches only touch orchestration (GPU and device pooling), not the engine.

✅ Check 11 (Golden AL): PASS — The launcher adds roles.agg.args.spec-decode-acceptance-length=3.39 (seen in the c1 srt-single-node-submission.json). This equals the golden qwen3.5-397b-a17b-nvfp4__mtp__native__default-default value for MTP with 3 draft tokens and thinking on.

➖ Check 12 (Append-only): N/A — The new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — Checkpoint 410f915f excludes every mtp.* module (fc, attention, shared expert, 512 experts, gate) from quantization. Qwen3_5MTP.__init__ re-indexes those excludes to layer 60, so the draft loads in BF16 as shipped. The draft KV cache is FP8, the same as the target's. Nothing enables SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE (ATOM path).

✅ Check 14 (Pareto coverage): PASS — Curve qwen3.5 / agentic-coding / mi355x_atom / fp4 / P90 E2EL from run 42460635358 (source 3fd52b00, bmk_agentic_* artifacts): 10 of 11 measured points are on the frontier (pareto_coverage PASS). Only c1 is dominated.

➖ Check 16 (NVIDIA overview): N/A — MI355X only; no NVIDIA configuration is affected.

➖ Check 17 (Metrics publication): N/A — ATOM serving does not use the Dynamo TRT-LLM metrics controls.

✅ Check 18 (Recipe/image/sweep consistency): PASS — (a) All 11 points generate with no errors. (b) TP4 on 4 GPUs, and the conc lists map 1:1 to the overrides; 4 x 253 GiB stays within the 1,499 GB share. (c) The nightly_202611071509 tag and the sha256:39ac8572 digest both resolve. (d, e) Not applicable. (f) The changelog matches the recipe, including the shared state pool. (g) Nothing was removed. (i) ATOM streams one chunk per engine step, which is no more than 20 tokens.

Assessed commit: f4b5e7ee50ba07b2d0081cd19565f208b9d96e6a.

@seungrokj

Copy link
Copy Markdown
Collaborator

@zhuyuhua-v is this perf expected and can be merged as it is ?

@zhuyuhua-v

Copy link
Copy Markdown
Collaborator Author

@zhuyuhua-v is this perf expected and can be merged as it is ?

@seungrokj yes, please help merge this one

…x-atom-agentx-1018

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
…h TP2 points and INT4 quick all-reduce

- Image rocm/atom-dev:nightly_202611131030 (ATOM 0.1.10.dev122+g9fec99786, AITER 43840aee7; digest sha256:91c7d38b) carries ROCm/aiter PR 6981 and ROCm/ATOM PRs 3106 and 3108.
- Follow the MXFP4 submission configuration in ROCm/ATOM PR 3255: add TP2 points at CONC 12 on the GPU prefix cache and CONC 16, 20, 24, 32 and 40 with LMCache DRAM offload (same per-rank pools as TP4; 749 GB TP2 share), delayer caps 15 ticks from CONC 32.
- Set AITER_QUICK_REDUCE_QUANTIZATION=INT4 on every point, as the FP8 sibling does; qwen3.5 is grandfathered under the MODELS.md all-reduce policy.
@zhuyuhua-v zhuyuhua-v changed the title feat(agentx): add Qwen3.5 FP4 MI355X ATOM AgentX on AttnFP8-V2 and nightly_202611071509 feat(agentx): add Qwen3.5 FP4 MI355X ATOM AgentX (TP4 + TP2) on AttnFP8-V2 and nightly_202611131030 Nov 13, 2026
@zhuyuhua-v zhuyuhua-v added AMD full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures agentx AgentX benchmarks, recipes, and infrastructure and removed AMD full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures agentx AgentX benchmarks, recipes, and infrastructure labels Nov 13, 2026
@zhuyuhua-v

Copy link
Copy Markdown
Collaborator Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure AMD full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants