Repository navigation
feat(agentx): add GLM-5.2 MI355X ATOM PD disaggregation points / feat(agentx): 新增 GLM-5.2 MI355X ATOM PD 分离式测试点 - #4045
Conversation
Add glm5.2-fp4-mi355x-atom-disagg-agentic-mtp: eight AgentX points with a CPP4 (PP4 x TP1) ATOM prefill behind AToMesh and Mooncake RDMA to the decodes. 1P1D on one node at C56-C112 keeps the prefill's prefix tier in a Mooncake Store of four per-NIC pools on reserved 2 MiB HugeTLB pages; 1P2D at C56 and 1P3D at C12-C24 use LMCache in the prefill. host_setup drops the page cache and compacts memory before every point. Ship NVIDIA/srt-slurm#604 as runners/srt-slurm/patches/atom-tp-from-pp.patch so srtctl launches the PP4 prefill with -tp 1. 新增 glm5.2-fp4-mi355x-atom-disagg-agentic-mtp:八个 AgentX 测试点。ATOM prefill 为 CPP4(PP4 x TP1),经 AToMesh 路由,并通过 Mooncake RDMA 把 KV 传给 decode。单节点 1P1D(C56-C112)把 prefill 的前缀缓存层放在 Mooncake Store 中:四个按网卡划分的池,使用预留的 2 MiB HugeTLB 大页;1P2D(C56)和 1P3D(C12-C24)在 prefill 中使用 LMCache。每个测试点开始前,host_setup 都会 清理页缓存并整理内存。 以 runners/srt-slurm/patches/atom-tp-from-pp.patch 的形式附带 NVIDIA/srt-slurm#604,使 srtctl 以 -tp 1 启动 PP4 prefill。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
functionstackx
left a comment
There was a problem hiding this comment.
u might need to edit inferencex-e2e/infx/results/agentic/server_metrics.py & inferencex-e2e/infx/results/agentic/backends/atom.py for atom disagg too for it to collect mooncake prom metrics https://github.com/SemiAnalysisAI/InferenceX/pull/3964/changes
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it's a large, complex multi-node benchmark change — new ATOM prefill/decode-disaggregation recipe variants, Mooncake/LMCache service orchestration, hugepage reservation logic, and a backend patch to ATOM's tensor-parallel sizing — a human look would still be worthwhile.
What was reviewed:
- The new
atom-tp-from-pp.patch: it only changes-tpsizing whenpipeline-parallel-size/prefill-context-parallel-size> 1, and no existing ATOM recipe in the repo sets those, so prior ATOM points are unaffected. perf-changelog.yaml: confirmed the diff is a pure append (12 new lines at the file's tail, no earlier bytes touched) and the entry is English-only.- Cross-checked tp/pp/dcp-size settings between
amd-master.yaml's search space anddisagg-variants.yaml's role definitions — they match. - The ruled-out candidate issue (possible eval-result overwrite risk, as the first pp>1 multinode agentic entry) is worth a maintainer's confirmation given it wasn't run before this PR.
Extended reasoning...
The diff adds a new multi-node GLM-5.2 ATOM PD-disaggregation benchmark recipe (335 lines of YAML with Mooncake/LMCache service orchestration, hugepage reservation host_setup/teardown, NUMA-pinned RDMA services), wires it into amd-master.yaml, appends one perf-changelog.yaml block, and includes a vendored srt-slurm patch that changes how ATOM's backend computes -tp from GPU count/pipeline-parallel-size. No auth/crypto/permissions surface is touched; the sensitive bits are correctness of resource math (GPU/hugepage counts, port assignments) and whether the new -tp derivation could regress other ATOM configs — I confirmed no other ATOM recipe currently sets pipeline-parallel-size, so the patch is a no-op elsewhere. Size and multi-service complexity, plus the PR's own note that this has not yet been run on the target cluster, are why a human should still look even though no bugs were found.
This review covers commit 8df9856, which is no longer the latest commit on this pull request; later commits are not covered by it.
…om-disagg-agentx # Conflicts: # inferencex-e2e/runners/srt-slurm/patches/README.md
…es above held pages - Set HF_HOME/HF_HUB_CACHE on the ATOM roles. amd/GLM-5.2-MXFP4 has no staged copy on mi355x-amds, so the job serves hf:; srtctl downloads it once into the shared hub cache only when a role sets HF_HOME. - Follow the UMBP recipe's host_setup for the Store: the cluster setup.sh frees leftover hugepages and drops the page cache, then the prefill node reserves the owners' 714752 2 MiB pages plus 2% on top of the pages host services hold. mi355x-amds nodes hold 5656, which the old fixed target of 716800 left too few for both NUMA nodes. The LMCache points keep the cluster default host setup. - Run the four Store points from one variant, so the planner schedules one GSM8K eval for them instead of four. - Drop header and section comments per AGENTS.md, and rename the srt-slurm patch to 604-atom-tp-from-pp.patch like the other patches. - 为 ATOM 角色设置 HF_HOME/HF_HUB_CACHE。mi355x-amds 上没有预置的 amd/GLM-5.2-MXFP4,任务以 hf: 方式加载;只有角色设置了 HF_HOME,srtctl 才会把模型一次性下载到共享 hub 缓存。 - Store 的 host_setup 参照 UMBP recipe:先由集群 setup.sh 释放残留大页并 清理页缓存,再在 prefill 节点上于主机服务已占用的大页之上,预留 owner 所需 的 714752 个 2 MiB 页并多留 2%。mi355x-amds 节点常驻占用 5656 页,原先固定 的 716800 目标会导致两个 NUMA 节点的空闲页不足。LMCache 测试点沿用集群默认 的 host setup。 - 四个 Store 测试点共用一个变体,规划器只为它们安排一个 GSM8K eval,而不是 四个。 - 按 AGENTS.md 删除头注释与段落注释,并把 srt-slurm 补丁按惯例重命名为 604-atom-tp-from-pp.patch。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ache Mount /var/lib/hf-hub-cache at /mnt/hf_hub_cache and point HF_HUB_CACHE at it, as the single-node GLM-5.2 points do, so nodes that already hold amd/GLM-5.2-MXFP4 load it from local NVMe instead of downloading it into the shared cache. HF_HOME stays set so srtctl fills the first node's cache before workers start. 像单机 GLM-5.2 测试点一样,把 /var/lib/hf-hub-cache 挂载到 /mnt/hf_hub_cache,并让 HF_HUB_CACHE 指向它。已经有 amd/GLM-5.2-MXFP4 的 节点会直接从本地 NVMe 加载,不再下载到共享缓存。保留 HF_HOME,使 srtctl 在 worker 启动前先补齐第一个节点的缓存。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On mi355x-amds both Store points failed in the first owner: every ibv_reg_mr of its 349 GiB HugeTLB segment returned ENOMEM at once. The pyxis --container-remap-root user is root only inside its namespace, so the NIC's pin of the segment counts against RLIMIT_MEMLOCK. Run `ulimit -l unlimited` before each owner starts, as the DSv4 SGLang GB200 and GB300 recipes do for their Mooncake stores. 两个 Store 测试点都在 mi355x-amds 上的第一个 owner 处失败:它对 349 GiB HugeTLB 段的每次 ibv_reg_mr 都立刻返回 ENOMEM。pyxis 的 --container-remap-root 用户只在自己的命名空间内是 root,网卡锁定该段时会 计入 RLIMIT_MEMLOCK。现在每个 owner 启动前先执行 `ulimit -l unlimited`, 与 DSv4 SGLang GB200、GB300 配方对 Mooncake store 的做法一致。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (cherry-pick #3964) Carry #3964 (2db0192, 0fda3f1, 0842d82) so this PR's sweep prices the ATOM P/D cache rates with it: count only prefill endpoints, take the CPU rate from atom:prefix_cache_offload_tokens (reported for LMCache and the Mooncake Store alike), and slice counters to the profiling window. Applied as #3964's net diff onto current main; drop it once #3964 merges. 携带 #3964(2db01926、0fda3f1e、0842d829),使本 PR 的 sweep 用它来计算 ATOM P/D 的缓存命中率:只统计 prefill 端点,CPU 命中率取自 atom:prefix_cache_offload_tokens(LMCache 与 Mooncake Store 都会上报), 并按 profiling 窗口切分计数。以 #3964 相对其基线的净改动应用到当前 main 上; #3964 合入后即可移除。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Thanks! Cherry-picked #3964 into this PR so the sweep uses the fixed ATOM P/D cache accounting. The Mooncake Store needs no separate scrape: ATOM reports its reuse in the same |
srtctl's model pre-download runs without its bash wrapper, so it sees only the HF_HOME it exports itself, not the role's HF_HUB_CACHE. On the c72 Store point it therefore pulled all 438 GB into the container's /tmp/hf_home/hub while the workers loaded the copy already in /var/lib/hf-hub-cache. Mount that cache at $HF_HOME/hub so the pre-download checks, and if needed fills, the same directory the workers read. srtctl 的模型预下载不经过 bash 包装,只能看到它自己 export 的 HF_HOME, 看不到角色环境里的 HF_HUB_CACHE。因此在 Store c72 测试点上,它把整整 438 GB 下载到了容器内的 /tmp/hf_home/hub,而 worker 实际加载的是 /var/lib/hf-hub-cache 中已有的副本。现在把该缓存挂载到 $HF_HOME/hub, 使预下载检查(必要时填充)的正是 worker 读取的同一目录。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42972857386 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42972857386 |
…om-disagg-agentx Resolve the perf-changelog conflict with prepare_perf_changelog_merge resolve-conflict, which keeps this PR's entry as the tail append, and keep both rows in runners/srt-slurm/patches/README.md: 602 from main and 604 from this PR. All three patches apply in order on srt-slurm 545233a0d. 用 prepare_perf_changelog_merge resolve-conflict 解决 perf-changelog 冲突, 本 PR 的条目仍是末尾追加;runners/srt-slurm/patches/README.md 保留 main 的 602 和本 PR 的 604 两行。三个补丁按顺序都能打到 srt-slurm 545233a0d 上。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3c61360 to
767819a
Compare
…om-disagg-agentx Resolve the perf-changelog conflict with #4071 using prepare_perf_changelog_merge resolve-conflict; this PR's entry stays the tail append. 用 prepare_perf_changelog_merge resolve-conflict 解决与 #4071 的 perf-changelog 冲突,本 PR 的条目仍是末尾追加。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om-disagg-agentx Resolve the perf-changelog conflict with #4050 using prepare_perf_changelog_merge resolve-conflict; this PR's entry stays the tail append. 用 prepare_perf_changelog_merge resolve-conflict 解决与 #4050 的 perf-changelog 冲突,本 PR 的条目仍是末尾追加。 [skip-sweep] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only conflict is perf-changelog.yaml: main's file with this PR's entry appended at the tail (prepare_perf_changelog_merge resolve-conflict). 合并 main。唯一的冲突是 perf-changelog.yaml:采用 main 的文件,并把本 PR 的条目 追加在末尾(prepare_perf_changelog_merge resolve-conflict)。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ages / Mooncake Store owner 改用 THP,不再预留 HugeTLB 页 rocm/atom-dev:nightly_202611110332 builds Mooncake 206d634, which includes kvcache-ai/Mooncake#5151: an rdma Store owner maps its segment 2 MiB aligned with MADV_HUGEPAGE on the NUMA node numa_exec binds it to. The owners therefore drop MC_STORE_USE_HUGEPAGE, the variants drop their host_setup, and the mi355x-amds hook loses its HEAD_NODE_HUGEPAGES_PER_NUMA block. The cluster's default host setup still drops the page cache for kv-offloading dram before any service starts, which leaves the free 2 MiB blocks the segments need. amd-master moves to that image and records the ATOM (router) and Mooncake source commits it ships. rocm/atom-dev:nightly_202611110332 编译的是 Mooncake 206d634,其中包含 kvcache-ai/Mooncake#5151:rdma 的 Store owner 在 numa_exec 绑定的 NUMA 节点上 以 2 MiB 对齐映射 segment,并加 MADV_HUGEPAGE。因此 owner 去掉 MC_STORE_USE_HUGEPAGE,各 variant 去掉 host_setup,mi355x-amds hook 去掉 HEAD_NODE_HUGEPAGES_PER_NUMA 这段。kv-offloading 为 dram 时,集群默认的 host setup 仍会在所有服务启动前 drop page cache,留出 segment 需要的空闲 2 MiB 块。 amd-master 换成该镜像,并记录它所带的 ATOM(router)和 Mooncake 源码 commit。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@functionstackx This is now fixed in Mooncake: kvcache-ai/Mooncake#5151 (merged as 206d634) makes rdma Store owners request THP with The PR now relies on it and reserves no hugepages:
The full re-sweep is running: run 42972857386. |
|
Thanks @Jasen2201 it seems like using madvise 2MB hugepages ends up working. Waiting for sweep to pass |
Resolves the perf-changelog conflict only: main's new entries come first and this PR's entry stays at the tail, unchanged. The sweep of this config, 42972857386 on 00005f5, remains the run to reuse: - The recipe, the master config entry and the changelog entry are byte-identical to 00005f5, and `infx generate` gives the same 9 recipes and manifest, fingerprints included. - main 68d69ac..88a44f1 reaches these jobs' launch path in two places, and neither changes them: runners.yaml adds a Kimi-K3.1-Pro model-cache entry for mi355x-amds, and infx/matrix/plan.py adds --repository / --pull-request filtering for push-to-main changelog metadata. The rest is CI-bot workflows, other models' recipes and the pretraining/torchtitan submodule. The srt-slurm pin and benchmark-multinode-tmpl.yml are unchanged. 只解决 perf-changelog 冲突:main 的新条目在前,本 PR 的条目仍在末尾且未改动。 该配置的扫描 42972857386(00005f531)仍是要复用的运行: - 配方、master 配置条目和 changelog 条目与 00005f5 逐字节相同, `infx generate` 生成的 9 个配方和 manifest(含指纹)完全一致。 - main 68d69ac..88a44f1 只有两处触及这些作业的启动路径,均不改变作业: runners.yaml 为 mi355x-amds 新增 Kimi-K3.1-Pro 的模型缓存条目; infx/matrix/plan.py 为推送到 main 的 changelog 元数据新增 --repository / --pull-request 过滤。其余改动是 CI 机器人工作流、其他模型的 配方和 pretraining/torchtitan 子模块。srt-slurm 固定版本和 benchmark-multinode-tmpl.yml 均未改变。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/reuse-sweep-run 42972857386 |
|
/use 42972857386 |
|
@seungrokj staged run 42972857386. This run remains available across future |
…e diff - Add GLM-5.2 MXFP4 on MI355X with ATOM prefill/decode disaggregation behind AToMesh, Mooncake RDMA transfer and MTP, on rocm/atom-dev:nightly_202611110332 (ATOM 920a200a3, Mooncake 206d634), pinned by digest. - Every point has one PP4 x TP1 prefill (layers 20/20/20/18) with a Mooncake Store prefix tier in DRAM: 1P1D with a colocated TP4 DCP4 decode at concurrency 56, 72, 96, 112 and 128; 1P2D with two TP4 DCP4 decodes at 56; 1P3D with three TP4 decodes at 12, 16 and 24. - MTP draft length 3 for 1P1D, 4 for 1P2D and 5 for 1P3D. 1P1D shares one Store pool (2 x 768 GiB owners); 1P2D and 1P3D use a pool per prefill NIC (4 x 349 GiB) and cache-aware routing. dram-utilization is 1.0. - Mooncake 206d634 includes kvcache-ai/Mooncake#5151: the RDMA Store owners back their segments with THP through madvise, so no hugepages are reserved. - Weights run as shipped in amd/GLM-5.2-MXFP4: MXFP4 routed and shared experts in layers 3-77, BF16 elsewhere, with no online quantization flag. ATOM's default MLA absorb runs the kv_b_proj up-projections as FP8 batched GEMMs. All-reduce is BF16 (AITER quick reduce unset, default NONE) and the KV and indexer caches are FP8. - No change to the MTP draft (model.layers.78): it is stored in BF16, including its routed and shared experts, and is served through the image defaults with no draft override, so its kv_b_proj takes the same FP8 MLA absorb as the target and it shares the target's FP8 KV and indexer caches. - 在 MI355X 上新增 GLM-5.2 MXFP4 的 ATOM prefill/decode 分离部署,使用 AToMesh 路由、Mooncake RDMA 传输与 MTP,镜像为 rocm/atom-dev:nightly_202611110332(ATOM 920a200a3、Mooncake 206d634),按 digest 固定。 - 每个点都有一个 PP4 x TP1 prefill(各级层数 20/20/20/18),并在 DRAM 中使用 Mooncake Store 前缀缓存层:1P1D 为同机 TP4 DCP4 decode,并发 56、72、96、112、128;1P2D 为两个 TP4 DCP4 decode,并发 56;1P3D 为三个 TP4 decode,并发 12、16、24。 - MTP 草稿长度:1P1D 为 3,1P2D 为 4,1P3D 为 5。1P1D 共享一个 Store 池(2 x 768 GiB owner);1P2D 与 1P3D 每个 prefill 网卡一个池(4 x 349 GiB),并使用缓存感知路由。dram-utilization 为 1.0。 - Mooncake 206d634 包含 kvcache-ai/Mooncake#5151:RDMA Store owner 通过 madvise 以 THP 承载其内存段,因此不预留 hugepages。 - 权重按 amd/GLM-5.2-MXFP4 原样运行:第 3-77 层的路由专家与共享专家为 MXFP4,其余为 BF16,未设置在线量化参数。ATOM 默认的 MLA absorb 将 kv_b_proj 上投影以 FP8 批量 GEMM 运行。All-reduce 为 BF16(未设置 AITER quick reduce,默认 NONE),KV 缓存与 indexer 缓存为 FP8。 - MTP 草稿(model.layers.78)无变化:以 BF16 存储(包括其路由专家与共享专家),通过镜像默认设置服务、无草稿覆盖,因此其 kv_b_proj 与目标模型一样走默认的 FP8 MLA absorb,并共享目标模型的 FP8 KV 与 indexer 缓存。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
checked again and edited #4045 (comment) |
Co-authored-by: Perplexity <noreply@perplexity.ai>
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 10, rawBenchmarkRows 18, coveredByPerJobFiles 9, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 0, powerPublicationPoints 9, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Summary
Adds
glm5.2-fp4-mi355x-atom-disagg-agentic-mtp: nine GLM-5.2 MXFP4 AgentX points on MI355X with ATOM prefill/decode disaggregation behind AToMesh, Mooncake RDMA from prefill to decode, MTP, and the prefill's prefix cache in a Mooncake Store. The single-node ATOM entry stops at concurrency 48; these points cover concurrency 12-128 on one to four nodes.Every point has one CPP4 prefill worker (PP4 x TP1 on GPUs 0-3, layers split 20/20/20/18):
rocm/atom-dev:nightly_202611110332, pinned by digest (ROCm/ATOM 920a200a3, Mooncake 206d634). Weights run as shipped inamd/GLM-5.2-MXFP4: routed and shared experts in layers 3-77 are MXFP4 and everything else is BF16, with no online quantization flag. ATOM's default MLA absorb (ATOM_MLA_ABSORB_BF16=0) quantizes thekv_b_projup-projections to FP8 at load and runs them as FP8 batched GEMMs. All-reduce is BF16 (AITER quick reduce unset, defaultNONE); the KV and indexer caches are FP8.genericservices on the prefill node. 1P1D, like ATOM's CI, has one master and two 768 GiB owners, one per NUMA node, shared by all four PP stages; each owner registers on rdma0-3, and the prefill setsMC_ENABLE_DEST_DEVICE_AFFINITYso each stage reaches an owner through the NIC of the same name. 1P2D and 1P3D start a master and a 349 GiB owner for each prefill-stage NIC (rdma0-3), so each stage reads its own pool through its own NIC.madvise(MADV_HUGEPAGE), on the NUMA node thatnuma_exec.pybinds it to. This PR reserves no hugepages and changes no cluster hook. The cluster's default host setup drops the page cache forkv-offloading: drambefore any service starts, which leaves the free 2 MiB blocks the segments need. Owners runulimit -l unlimitedfirst: the remapped container root lacks hostCAP_IPC_LOCK, so registering the segment counts againstRLIMIT_MEMLOCK.dram-utilization: 1.0gives each 4-GPU server 1499 GB. The 1P1D decode shares the prefill's node, so its owners hold 1649 GB, 825 GB per NUMA node, within the prefill's and the decode's shares. The 1P2D and 1P3D decodes run on their own nodes, so those owners hold the prefill's 1499 GB.amd/GLM-5.2-MXFP4has no staged copy onmi355x-amds; as the single-node GLM-5.2 points do, workers read it from each node's local hub cache (/var/lib/hf-hub-cachemounted at$HF_HOME/hub, the only path srtctl's pre-download sees), and srtctl fills the first node's cache before they start. The five 1P1D points share one recipe variant, so the sweep runs one GSM8K eval for them.-tpas the worker's GPUs divided by its pipeline and prefill-context stages (fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604). main's srt-slurm pin includes it, so this PR carries no srt-slurm patch.infx. The ATOM P/D cache rates come from the counters already inserver_metrics_export.json, which fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 reads at ingest: prefill endpoints only, withatom:prefix_cache_offload_tokensas the CPU tier.model.layers.78ofamd/GLM-5.2-MXFP4) ships all BF16, including its 256 routed and shared experts. Both servers logonline_quant_config: None, and the recipe sets no draft override, so it is served through the image defaults: like the target, itskv_b_projtakes ATOM's default FP8 MLA absorb. It shares the target's FP8 KV cache and FP8 indexer cache: the recipe sets noindex_cache_dtype, so the indexer followskv_cache_dtype.kv_b_projin FP8. Quick-reduce INT4, PTPC online quantization, index sharing, FP4 KV and SSM state cache are not used. GSM8K is within the cohort bar. Details: draft data type and accuracy comment.Runs on
mi355x-amds. Sweep 42972857386 ran every point and eval green on the head00005f531, with 0 failed requests. Before it, a 1P1D C112 preflight (42959567550) checked the THP owners: both 768 GiB owners loggedAllocated THP bufferand registered in about 15 s.AI model disclosure
claude-opus-5-5(Claude Opus 5.5), running in Claude Code.claude-opus-5-5(Claude Opus 5.5), running in Claude Code for @seungrokj, rewrote the perf-changelog block, checked the draft data types and accuracy-sensitive optimizations at ATOM 920a200a3, and corrected the precision lines of this description.PR checklist and change type
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概要
新增
glm5.2-fp4-mi355x-atom-disagg-agentic-mtp:MI355X 上的九个 GLM-5.2 MXFP4 AgentX 测试点,采用 ATOM 预填充/解码分离(由 AToMesh 路由),预填充到解码用 Mooncake RDMA 传输,启用 MTP,并把预填充的前缀缓存放在 Mooncake Store 中。单机 ATOM 条目最高到并发 48;这些测试点在一到四个节点上覆盖并发 12-128。每个测试点都有一个 CPP4 预填充 worker(GPU 0-3 上 PP4 x TP1,层划分为 20/20/20/18):
rocm/atom-dev:nightly_202611110332,按 digest 固定(ROCm/ATOM 920a200a3,Mooncake 206d634)。权重按amd/GLM-5.2-MXFP4原样运行:第 3-77 层的路由专家与共享专家为 MXFP4,其余为 BF16,未设置在线量化参数。ATOM 默认的 MLA absorb(ATOM_MLA_ABSORB_BF16=0)在加载时把kv_b_proj上投影量化为 FP8,并以 FP8 批量 GEMM 运行。all-reduce 为 BF16(未设置 AITER quick reduce,默认NONE);KV 缓存与 indexer 缓存为 FP8。genericservice 运行在预填充节点上。1P1D 与 ATOM CI 一致,使用一个 master 和两个 768 GiB owner(每个 NUMA 节点一个),由四个 PP stage 共享;每个 owner 注册在 rdma0-3 上,预填充设置MC_ENABLE_DEST_DEVICE_AFFINITY,使每个 stage 通过同名网卡访问 owner。1P2D 和 1P3D 为每个预填充 stage 的网卡(rdma0-3)各启动一个 master 和一个 349 GiB owner,使每个 stage 通过自己的网卡读取自己的池。numa_exec.py绑定的 NUMA 节点上以 2 MiB 对齐映射 segment,并调用madvise(MADV_HUGEPAGE)。本 PR 不预留大页,也不修改集群 hook。kv-offloading: dram时,集群默认的 host setup 会在任何服务启动前清理页缓存,为 segment 留出所需的空闲 2 MiB 块。owner 先执行ulimit -l unlimited:重映射的容器 root 没有主机的CAP_IPC_LOCK,注册 segment 会计入RLIMIT_MEMLOCK。dram-utilization: 1.0为每台 4-GPU 服务器分配 1499 GB。1P1D 的解码与预填充共用节点,因此其 owner 共占 1649 GB,每个 NUMA 节点 825 GB,在预填充和解码各自的份额之内。1P2D 和 1P3D 的解码运行在独立节点上,因此 owner 占用预填充的 1499 GB。mi355x-amds上没有预置的amd/GLM-5.2-MXFP4;与单机 GLM-5.2 测试点一样,worker 从各节点本地的 hub 缓存读取模型(/var/lib/hf-hub-cache挂载到$HF_HOME/hub,这是 srtctl 预下载唯一能看到的路径),srtctl 会在 worker 启动前先补齐第一个节点的缓存。五个 1P1D 测试点共用一个配方变体,因此扫描只为它们运行一个 GSM8K eval。-tp设为 worker 的 GPU 数除以其流水线并行和预填充上下文并行的 stage 数(fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604)。main 固定的 srt-slurm 版本已包含该改动,因此本 PR 不带 srt-slurm 补丁。infx。ATOM P/D 的缓存命中率来自server_metrics_export.json中已有的计数器,fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 在导入时读取:只统计预填充端点,CPU 层为atom:prefix_cache_offload_tokens。amd/GLM-5.2-MXFP4的model.layers.78)发布时全部为 BF16,包括其 256 个路由专家与共享专家。两个服务端的日志都显示online_quant_config: None,配方也未设置草稿覆盖,因此按镜像默认处理:与目标模型一样,其kv_b_proj走 ATOM 默认的 FP8 MLA absorb。它与目标模型共用 FP8 KV 缓存和 FP8 indexer 缓存:配方未设置index_cache_dtype,indexer 跟随kv_cache_dtype。kv_b_proj。Quick-reduce INT4、PTPC 在线量化、index sharing、FP4 KV 与 SSM 状态缓存均未使用。GSM8K 在 cohort 阈值以内。详见草稿数据类型与精度评论。在
mi355x-amds上的运行。 扫描 42972857386 在 head00005f531上全部测试点和 eval 均通过,失败请求数为 0。在此之前,一次 1P1D C112 预检(42959567550)验证了 THP owner:两个 768 GiB owner 都打印了Allocated THP buffer,并在约 15 秒内完成注册。AI 模型使用说明
claude-opus-5-5(Claude Opus 5.5),在 Claude Code 中运行。claude-opus-5-5(Claude Opus 5.5)在 Claude Code 中代 @seungrokj 重写了 perf-changelog 条目,在 ATOM 920a200a3 上核查了草稿数据类型与精度敏感优化,并修正了本描述中的精度说明。改动类型
新功能;配置变更。
🤖 Generated with Claude Code