Skip to content

feat(agentx): add GLM-5.2 MI355X ATOM PD disaggregation points / feat(agentx): 新增 GLM-5.2 MI355X ATOM PD 分离式测试点 - #4045

Merged
edwingao28 merged 54 commits into
mainfrom
jasen/glm52-mi355x-atom-disagg-agentx
Nov 11, 2026
Merged

edwingao28 merged 54 commits into
mainfrom
jasen/glm52-mi355x-atom-disagg-agentx

Conversation

@Jasen2201

@Jasen2201 Jasen2201 commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds glm5.2-fp4-mi355x-atom-disagg-agentic-mtp: nine GLM-5.2 MXFP4 AgentX points on MI355X with ATOM prefill/decode disaggregation behind AToMesh, Mooncake RDMA from prefill to decode, MTP, and the prefill's prefix cache in a Mooncake Store. The single-node ATOM entry stops at concurrency 48; these points cover concurrency 12-128 on one to four nodes.

Every point has one CPP4 prefill worker (PP4 x TP1 on GPUs 0-3, layers split 20/20/20/18):

Points Nodes Decode MTP Mooncake Store on the prefill node
C56, C72, C96, C112, C128 1 (1P1D) TP4 + DCP4 on GPUs 4-7 of the prefill node 3 one shared pool, 2 x 768 GiB
C56 3 (1P2D) 2 x TP4 + DCP4, a node each 4 a pool per NIC, 4 x 349 GiB
C12, C16, C24 4 (1P3D) 3 x TP4, a node each 5 a pool per NIC, 4 x 349 GiB
  • Image and precision. rocm/atom-dev:nightly_202611110332, pinned by digest (ROCm/ATOM 920a200a3, Mooncake 206d634). Weights run as shipped in amd/GLM-5.2-MXFP4: routed and shared experts in layers 3-77 are MXFP4 and everything else is BF16, with no online quantization flag. ATOM's default MLA absorb (ATOM_MLA_ABSORB_BF16=0) quantizes the kv_b_proj up-projections to FP8 at load and runs them as FP8 batched GEMMs. All-reduce is BF16 (AITER quick reduce unset, default NONE); the KV and indexer caches are FP8.
  • Mooncake Store. The masters and owners run as generic services on the prefill node. 1P1D, like ATOM's CI, has one master and two 768 GiB owners, one per NUMA node, shared by all four PP stages; each owner registers on rdma0-3, and the prefill sets MC_ENABLE_DEST_DEVICE_AFFINITY so each stage reaches an owner through the NIC of the same name. 1P2D and 1P3D start a master and a 349 GiB owner for each prefill-stage NIC (rdma0-3), so each stage reads its own pool through its own NIC.
  • Store memory. An owner registers its segment with the NIC in 64 GiB regions, and the NIC maps a region only when it is all 2 MiB pages. Mooncake 206d634 includes [Store] Request THP for base-page RDMA global segments kvcache-ai/Mooncake#5151: an rdma owner maps its segment 2 MiB-aligned with madvise(MADV_HUGEPAGE), on the NUMA node that numa_exec.py binds it to. This PR reserves no hugepages and changes no cluster hook. The cluster's default host setup drops the page cache for kv-offloading: dram before any service starts, which leaves the free 2 MiB blocks the segments need. Owners run ulimit -l unlimited first: the remapped container root lacks host CAP_IPC_LOCK, so registering the segment counts against RLIMIT_MEMLOCK.
  • DRAM budget. With [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 #4211, dram-utilization: 1.0 gives each 4-GPU server 1499 GB. The 1P1D decode shares the prefill's node, so its owners hold 1649 GB, 825 GB per NUMA node, within the prefill's and the decode's shares. The 1P2D and 1P3D decodes run on their own nodes, so those owners hold the prefill's 1499 GB.
  • Model. amd/GLM-5.2-MXFP4 has no staged copy on mi355x-amds; as the single-node GLM-5.2 points do, workers read it from each node's local hub cache (/var/lib/hf-hub-cache mounted at $HF_HOME/hub, the only path srtctl's pre-download sees), and srtctl fills the first node's cache before they start. The five 1P1D points share one recipe variant, so the sweep runs one GSM8K eval for them.
  • srt-slurm. The PP4 prefill needs srtctl's ATOM backend to size -tp as the worker's GPUs divided by its pipeline and prefill-context stages (fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604). main's srt-slurm pin includes it, so this PR carries no srt-slurm patch.
  • Cache accounting. This PR does not change infx. The ATOM P/D cache rates come from the counters already in server_metrics_export.json, which fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 reads at ingest: prefill endpoints only, with atom:prefix_cache_offload_tokens as the CPU tier.
  • Draft data types: no change. The MTP draft (model.layers.78 of amd/GLM-5.2-MXFP4) ships all BF16, including its 256 routed and shared experts. Both servers log online_quant_config: None, and the recipe sets no draft override, so it is served through the image defaults: like the target, its kv_b_proj takes ATOM's default FP8 MLA absorb. It shares the target's FP8 KV cache and FP8 indexer cache: the recipe sets no index_cache_dtype, so the indexer follows kv_cache_dtype.
  • Accuracy-sensitive optimizations: two in use, recorded. Shared experts in layers 3-77 are MXFP4 in the AMD checkpoint (the trainer ships them in BF16), and ATOM's default MLA absorb runs kv_b_proj in FP8. Quick-reduce INT4, PTPC online quantization, index sharing, FP4 KV and SSM state cache are not used. GSM8K is within the cohort bar. Details: draft data type and accuracy comment.

Runs on mi355x-amds. Sweep 42972857386 ran every point and eval green on the head 00005f531, with 0 failed requests. Before it, a 1P1D C112 preflight (42959567550) checked the THP owners: both 768 GiB owners logged Allocated THP buffer and registered in about 15 s.

Point tok/s/GPU TTFT p50 / p90 ITL p50 GSM8K (strict)
1P1D C56 31,232 0.84 / 2.00 s 16.14 ms
1P1D C72 38,023 1.16 / 3.44 s 18.83 ms
1P1D C96 42,254 2.37 / 8.43 s 23.35 ms
1P1D C112 40,184 11.09 / 23.87 s 21.99 ms
1P1D C128 36,324 23.40 / 42.40 s 20.84 ms 0.967
1P2D C56 24,703 0.78 / 1.88 s 11.30 ms 0.967
1P3D C12 4,013 0.47 / 1.09 s 4.38 ms
1P3D C16 6,596 0.52 / 1.19 s 4.80 ms 0.971
1P3D C24 10,219 0.62 / 1.42 s 5.39 ms 0.968
AI model disclosure
  • Model/version: claude-opus-5-5 (Claude Opus 5.5), running in Claude Code.
  • Role: traced the InferenceX launcher and srt-slurm's ATOM backend; wrote the recipe, the master config entry, the perf-changelog entry and the srt-slurm patch that became fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604; moved the Store owners from reserved HugeTLB pages to THP after [Store] Request THP for base-page RDMA global segments kvcache-ai/Mooncake#5151 and removed the hugepage hook; ran the preflight and sweep 42972857386; resolved perf-changelog conflicts when merging main; and wrote this description.
  • Delegated agent: a Claude Code Explore subagent surveyed srt-slurm's ATOM backend and service placement. The runtime did not expose its model identifier, so it could not be verified.
  • Review sync: claude-opus-5-5 (Claude Opus 5.5), running in Claude Code for @seungrokj, rewrote the perf-changelog block, checked the draft data types and accuracy-sensitive optimizations at ATOM 920a200a3, and corrected the precision lines of this description.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of inferencex-e2e/perf-changelog.yaml and have not edited historical entries
  • If this PR can affect benchmark performance or adds or modifies a recipe, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

概要

新增 glm5.2-fp4-mi355x-atom-disagg-agentic-mtp:MI355X 上的九个 GLM-5.2 MXFP4 AgentX 测试点,采用 ATOM 预填充/解码分离(由 AToMesh 路由),预填充到解码用 Mooncake RDMA 传输,启用 MTP,并把预填充的前缀缓存放在 Mooncake Store 中。单机 ATOM 条目最高到并发 48;这些测试点在一到四个节点上覆盖并发 12-128。

每个测试点都有一个 CPP4 预填充 worker(GPU 0-3 上 PP4 x TP1,层划分为 20/20/20/18):

测试点 节点数 解码 MTP 预填充节点上的 Mooncake Store
C56、C72、C96、C112、C128 1(1P1D) 预填充节点 GPU 4-7 上的 TP4 + DCP4 3 一个共享池,2 x 768 GiB
C56 3(1P2D) 2 个 TP4 + DCP4,各占一个节点 4 每块网卡一个池,4 x 349 GiB
C12、C16、C24 4(1P3D) 3 个 TP4,各占一个节点 5 每块网卡一个池,4 x 349 GiB
  • 镜像与精度。 rocm/atom-dev:nightly_202611110332,按 digest 固定(ROCm/ATOM 920a200a3,Mooncake 206d634)。权重按 amd/GLM-5.2-MXFP4 原样运行:第 3-77 层的路由专家与共享专家为 MXFP4,其余为 BF16,未设置在线量化参数。ATOM 默认的 MLA absorb(ATOM_MLA_ABSORB_BF16=0)在加载时把 kv_b_proj 上投影量化为 FP8,并以 FP8 批量 GEMM 运行。all-reduce 为 BF16(未设置 AITER quick reduce,默认 NONE);KV 缓存与 indexer 缓存为 FP8。
  • Mooncake Store。 master 和 owner 均作为 generic service 运行在预填充节点上。1P1D 与 ATOM CI 一致,使用一个 master 和两个 768 GiB owner(每个 NUMA 节点一个),由四个 PP stage 共享;每个 owner 注册在 rdma0-3 上,预填充设置 MC_ENABLE_DEST_DEVICE_AFFINITY,使每个 stage 通过同名网卡访问 owner。1P2D 和 1P3D 为每个预填充 stage 的网卡(rdma0-3)各启动一个 master 和一个 349 GiB owner,使每个 stage 通过自己的网卡读取自己的池。
  • Store 内存。 owner 以 64 GiB 为单位向网卡注册 segment,网卡只有在区域全部由 2 MiB 页组成时才能映射。Mooncake 206d634 包含 [Store] Request THP for base-page RDMA global segments kvcache-ai/Mooncake#5151:rdma owner 在 numa_exec.py 绑定的 NUMA 节点上以 2 MiB 对齐映射 segment,并调用 madvise(MADV_HUGEPAGE)。本 PR 不预留大页,也不修改集群 hook。kv-offloading: dram 时,集群默认的 host setup 会在任何服务启动前清理页缓存,为 segment 留出所需的空闲 2 MiB 块。owner 先执行 ulimit -l unlimited:重映射的容器 root 没有主机的 CAP_IPC_LOCK,注册 segment 会计入 RLIMIT_MEMLOCK。
  • 内存预算。 依据 [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 #4211,dram-utilization: 1.0 为每台 4-GPU 服务器分配 1499 GB。1P1D 的解码与预填充共用节点,因此其 owner 共占 1649 GB,每个 NUMA 节点 825 GB,在预填充和解码各自的份额之内。1P2D 和 1P3D 的解码运行在独立节点上,因此 owner 占用预填充的 1499 GB。
  • 模型。 mi355x-amds 上没有预置的 amd/GLM-5.2-MXFP4;与单机 GLM-5.2 测试点一样,worker 从各节点本地的 hub 缓存读取模型(/var/lib/hf-hub-cache 挂载到 $HF_HOME/hub,这是 srtctl 预下载唯一能看到的路径),srtctl 会在 worker 启动前先补齐第一个节点的缓存。五个 1P1D 测试点共用一个配方变体,因此扫描只为它们运行一个 GSM8K eval。
  • srt-slurm。 PP4 预填充需要 srtctl 的 ATOM 后端把 -tp 设为 worker 的 GPU 数除以其流水线并行和预填充上下文并行的 stage 数(fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604)。main 固定的 srt-slurm 版本已包含该改动,因此本 PR 不带 srt-slurm 补丁。
  • 缓存统计。 本 PR 不修改 infx。ATOM P/D 的缓存命中率来自 server_metrics_export.json 中已有的计数器,fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 在导入时读取:只统计预填充端点,CPU 层为 atom:prefix_cache_offload_tokens。
  • 草稿模型数据类型:无变化。 MTP 草稿模型(amd/GLM-5.2-MXFP4 的 model.layers.78)发布时全部为 BF16,包括其 256 个路由专家与共享专家。两个服务端的日志都显示 online_quant_config: None,配方也未设置草稿覆盖,因此按镜像默认处理:与目标模型一样,其 kv_b_proj 走 ATOM 默认的 FP8 MLA absorb。它与目标模型共用 FP8 KV 缓存和 FP8 indexer 缓存:配方未设置 index_cache_dtype,indexer 跟随 kv_cache_dtype。
  • 精度敏感优化:两项在用,已记录。 AMD checkpoint 中第 3-77 层的共享专家为 MXFP4(训练方发布为 BF16),ATOM 默认的 MLA absorb 以 FP8 运行 kv_b_proj。Quick-reduce INT4、PTPC 在线量化、index sharing、FP4 KV 与 SSM 状态缓存均未使用。GSM8K 在 cohort 阈值以内。详见草稿数据类型与精度评论。

在 mi355x-amds 上的运行。 扫描 42972857386 在 head 00005f531 上全部测试点和 eval 均通过,失败请求数为 0。在此之前,一次 1P1D C112 预检(42959567550)验证了 THP owner:两个 768 GiB owner 都打印了 Allocated THP buffer,并在约 15 秒内完成注册。

测试点 tok/s/GPU TTFT p50 / p90 ITL p50 GSM8K(strict)
1P1D C56 31,232 0.84 / 2.00 s 16.14 ms
1P1D C72 38,023 1.16 / 3.44 s 18.83 ms
1P1D C96 42,254 2.37 / 8.43 s 23.35 ms
1P1D C112 40,184 11.09 / 23.87 s 21.99 ms
1P1D C128 36,324 23.40 / 42.40 s 20.84 ms 0.967
1P2D C56 24,703 0.78 / 1.88 s 11.30 ms 0.967
1P3D C12 4,013 0.47 / 1.09 s 4.38 ms
1P3D C16 6,596 0.52 / 1.19 s 4.80 ms 0.971
1P3D C24 10,219 0.62 / 1.42 s 5.39 ms 0.968

AI 模型使用说明

  • 模型/版本:claude-opus-5-5(Claude Opus 5.5),在 Claude Code 中运行。
  • 工作内容:梳理 InferenceX 启动器和 srt-slurm 的 ATOM 后端;编写配方、master 配置条目、perf-changelog 条目,以及后来成为 fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604 的 srt-slurm 补丁;在 [Store] Request THP for base-page RDMA global segments kvcache-ai/Mooncake#5151 之后把 Store owner 从预留的 HugeTLB 页改为 THP,并删除大页 hook;运行预检和扫描 42972857386;合并 main 时解决 perf-changelog 冲突;撰写本描述。
  • 委派的 agent:一个 Claude Code Explore 子 agent 调研了 srt-slurm 的 ATOM 后端和 service 放置方式。运行环境没有提供它的模型标识,因此无法确认。
  • 审阅同步:claude-opus-5-5(Claude Opus 5.5)在 Claude Code 中代 @seungrokj 重写了 perf-changelog 条目,在 ATOM 920a200a3 上核查了草稿数据类型与精度敏感优化,并修正了本描述中的精度说明。

改动类型

新功能;配置变更。

🤖 Generated with Claude Code

Add glm5.2-fp4-mi355x-atom-disagg-agentic-mtp: eight AgentX points with a
CPP4 (PP4 x TP1) ATOM prefill behind AToMesh and Mooncake RDMA to the
decodes. 1P1D on one node at C56-C112 keeps the prefill's prefix tier in a
Mooncake Store of four per-NIC pools on reserved 2 MiB HugeTLB pages; 1P2D
at C56 and 1P3D at C12-C24 use LMCache in the prefill. host_setup drops the
page cache and compacts memory before every point.

Ship NVIDIA/srt-slurm#604 as runners/srt-slurm/patches/atom-tp-from-pp.patch
so srtctl launches the PP4 prefill with -tp 1.

新增 glm5.2-fp4-mi355x-atom-disagg-agentic-mtp:八个 AgentX 测试点。ATOM
prefill 为 CPP4(PP4 x TP1),经 AToMesh 路由,并通过 Mooncake RDMA 把 KV
传给 decode。单节点 1P1D(C56-C112)把 prefill 的前缀缓存层放在 Mooncake
Store 中:四个按网卡划分的池,使用预留的 2 MiB HugeTLB 大页;1P2D(C56)和
1P3D(C12-C24)在 prefill 中使用 LMCache。每个测试点开始前,host_setup 都会
清理页缓存并整理内存。

以 runners/srt-slurm/patches/atom-tp-from-pp.patch 的形式附带
NVIDIA/srt-slurm#604,使 srtctl 以 -tp 1 启动 PP4 prefill。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

Set the new perf-changelog entry's pr-link to #4045.

将新增的 perf-changelog 条目的 pr-link 设为 #4045。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@functionstackx functionstackx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

u might need to edit inferencex-e2e/infx/results/agentic/server_metrics.py & inferencex-e2e/infx/results/agentic/backends/atom.py for atom disagg too for it to collect mooncake prom metrics https://github.com/SemiAnalysisAI/InferenceX/pull/3964/changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it's a large, complex multi-node benchmark change — new ATOM prefill/decode-disaggregation recipe variants, Mooncake/LMCache service orchestration, hugepage reservation logic, and a backend patch to ATOM's tensor-parallel sizing — a human look would still be worthwhile.

What was reviewed:

  • The new atom-tp-from-pp.patch: it only changes -tp sizing when pipeline-parallel-size/prefill-context-parallel-size > 1, and no existing ATOM recipe in the repo sets those, so prior ATOM points are unaffected.
  • perf-changelog.yaml: confirmed the diff is a pure append (12 new lines at the file's tail, no earlier bytes touched) and the entry is English-only.
  • Cross-checked tp/pp/dcp-size settings between amd-master.yaml's search space and disagg-variants.yaml's role definitions — they match.
  • The ruled-out candidate issue (possible eval-result overwrite risk, as the first pp>1 multinode agentic entry) is worth a maintainer's confirmation given it wasn't run before this PR.
Extended reasoning...

The diff adds a new multi-node GLM-5.2 ATOM PD-disaggregation benchmark recipe (335 lines of YAML with Mooncake/LMCache service orchestration, hugepage reservation host_setup/teardown, NUMA-pinned RDMA services), wires it into amd-master.yaml, appends one perf-changelog.yaml block, and includes a vendored srt-slurm patch that changes how ATOM's backend computes -tp from GPU count/pipeline-parallel-size. No auth/crypto/permissions surface is touched; the sensitive bits are correctness of resource math (GPU/hugepage counts, port assignments) and whether the new -tp derivation could regress other ATOM configs — I confirmed no other ATOM recipe currently sets pipeline-parallel-size, so the patch is a no-op elsewhere. Size and multi-service complexity, plus the PR's own note that this has not yet been run on the target cluster, are why a human should still look even though no bugs were found.

This review covers commit 8df9856, which is no longer the latest commit on this pull request; later commits are not covered by it.

Jasen2201 and others added 5 commits October 14, 2026 07:28
…om-disagg-agentx

# Conflicts:
#	inferencex-e2e/runners/srt-slurm/patches/README.md
…es above held pages

- Set HF_HOME/HF_HUB_CACHE on the ATOM roles. amd/GLM-5.2-MXFP4 has no
  staged copy on mi355x-amds, so the job serves hf:; srtctl downloads it
  once into the shared hub cache only when a role sets HF_HOME.
- Follow the UMBP recipe's host_setup for the Store: the cluster setup.sh
  frees leftover hugepages and drops the page cache, then the prefill node
  reserves the owners' 714752 2 MiB pages plus 2% on top of the pages host
  services hold. mi355x-amds nodes hold 5656, which the old fixed target of
  716800 left too few for both NUMA nodes. The LMCache points keep the
  cluster default host setup.
- Run the four Store points from one variant, so the planner schedules one
  GSM8K eval for them instead of four.
- Drop header and section comments per AGENTS.md, and rename the srt-slurm
  patch to 604-atom-tp-from-pp.patch like the other patches.

- 为 ATOM 角色设置 HF_HOME/HF_HUB_CACHE。mi355x-amds 上没有预置的
  amd/GLM-5.2-MXFP4,任务以 hf: 方式加载;只有角色设置了 HF_HOME,srtctl
  才会把模型一次性下载到共享 hub 缓存。
- Store 的 host_setup 参照 UMBP recipe:先由集群 setup.sh 释放残留大页并
  清理页缓存,再在 prefill 节点上于主机服务已占用的大页之上,预留 owner 所需
  的 714752 个 2 MiB 页并多留 2%。mi355x-amds 节点常驻占用 5656 页,原先固定
  的 716800 目标会导致两个 NUMA 节点的空闲页不足。LMCache 测试点沿用集群默认
  的 host setup。
- 四个 Store 测试点共用一个变体,规划器只为它们安排一个 GSM8K eval,而不是
  四个。
- 按 AGENTS.md 删除头注释与段落注释,并把 srt-slurm 补丁按惯例重命名为
  604-atom-tp-from-pp.patch。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ache

Mount /var/lib/hf-hub-cache at /mnt/hf_hub_cache and point HF_HUB_CACHE at
it, as the single-node GLM-5.2 points do, so nodes that already hold
amd/GLM-5.2-MXFP4 load it from local NVMe instead of downloading it into
the shared cache. HF_HOME stays set so srtctl fills the first node's cache
before workers start.

像单机 GLM-5.2 测试点一样,把 /var/lib/hf-hub-cache 挂载到
/mnt/hf_hub_cache,并让 HF_HUB_CACHE 指向它。已经有 amd/GLM-5.2-MXFP4 的
节点会直接从本地 NVMe 加载,不再下载到共享缓存。保留 HF_HOME,使 srtctl
在 worker 启动前先补齐第一个节点的缓存。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On mi355x-amds both Store points failed in the first owner: every
ibv_reg_mr of its 349 GiB HugeTLB segment returned ENOMEM at once. The
pyxis --container-remap-root user is root only inside its namespace, so
the NIC's pin of the segment counts against RLIMIT_MEMLOCK. Run
`ulimit -l unlimited` before each owner starts, as the DSv4 SGLang GB200
and GB300 recipes do for their Mooncake stores.

两个 Store 测试点都在 mi355x-amds 上的第一个 owner 处失败:它对 349 GiB
HugeTLB 段的每次 ibv_reg_mr 都立刻返回 ENOMEM。pyxis 的
--container-remap-root 用户只在自己的命名空间内是 root,网卡锁定该段时会
计入 RLIMIT_MEMLOCK。现在每个 owner 启动前先执行 `ulimit -l unlimited`,
与 DSv4 SGLang GB200、GB300 配方对 Mooncake store 的做法一致。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (cherry-pick #3964)

Carry #3964 (2db0192, 0fda3f1, 0842d82) so this PR's sweep prices the
ATOM P/D cache rates with it: count only prefill endpoints, take the CPU
rate from atom:prefix_cache_offload_tokens (reported for LMCache and the
Mooncake Store alike), and slice counters to the profiling window. Applied
as #3964's net diff onto current main; drop it once #3964 merges.

携带 #3964(2db01926、0fda3f1e、0842d829),使本 PR 的 sweep 用它来计算
ATOM P/D 的缓存命中率:只统计 prefill 端点,CPU 命中率取自
atom:prefix_cache_offload_tokens(LMCache 与 Mooncake Store 都会上报),
并按 profiling 窗口切分计数。以 #3964 相对其基线的净改动应用到当前 main 上;
#3964 合入后即可移除。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jasen2201

Copy link
Copy Markdown
Collaborator Author

u might need to edit inferencex-e2e/infx/results/agentic/server_metrics.py & inferencex-e2e/infx/results/agentic/backends/atom.py for atom disagg too for it to collect mooncake prom metrics https://github.com/SemiAnalysisAI/InferenceX/pull/3964/changes

Thanks! Cherry-picked #3964 into this PR so the sweep uses the fixed ATOM P/D cache accounting. The Mooncake Store needs no separate scrape: ATOM reports its reuse in the same atom:prefix_cache_offload_tokens counter as LMCache. I'll drop the cherry-pick once #3964 merges.

Jasen2201 and others added 2 commits October 14, 2026 09:36
srtctl's model pre-download runs without its bash wrapper, so it sees only
the HF_HOME it exports itself, not the role's HF_HUB_CACHE. On the c72
Store point it therefore pulled all 438 GB into the container's
/tmp/hf_home/hub while the workers loaded the copy already in
/var/lib/hf-hub-cache. Mount that cache at $HF_HOME/hub so the
pre-download checks, and if needed fills, the same directory the workers
read.

srtctl 的模型预下载不经过 bash 包装,只能看到它自己 export 的 HF_HOME,
看不到角色环境里的 HF_HUB_CACHE。因此在 Store c72 测试点上,它把整整
438 GB 下载到了容器内的 /tmp/hf_home/hub,而 worker 实际加载的是
/var/lib/hf-hub-cache 中已有的副本。现在把该缓存挂载到 $HF_HOME/hub,
使预下载检查(必要时填充)的正是 worker 读取的同一目录。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jasen2201 Jasen2201 added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

@Jasen2201 Jasen2201 added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Oct 15, 2026
…om-disagg-agentx

Resolve the perf-changelog conflict with prepare_perf_changelog_merge
resolve-conflict, which keeps this PR's entry as the tail append, and keep
both rows in runners/srt-slurm/patches/README.md: 602 from main and 604 from
this PR. All three patches apply in order on srt-slurm 545233a0d.

用 prepare_perf_changelog_merge resolve-conflict 解决 perf-changelog 冲突,
本 PR 的条目仍是末尾追加;runners/srt-slurm/patches/README.md 保留 main 的
602 和本 PR 的 604 两行。三个补丁按顺序都能打到 srt-slurm 545233a0d 上。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jasen2201
Jasen2201 force-pushed the jasen/glm52-mi355x-atom-disagg-agentx branch from 3c61360 to 767819a Compare October 15, 2026 02:59
…om-disagg-agentx

Resolve the perf-changelog conflict with #4071 using
prepare_perf_changelog_merge resolve-conflict; this PR's entry stays the
tail append.

用 prepare_perf_changelog_merge resolve-conflict 解决与 #4071 的
perf-changelog 冲突,本 PR 的条目仍是末尾追加。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om-disagg-agentx

Resolve the perf-changelog conflict with #4050 using
prepare_perf_changelog_merge resolve-conflict; this PR's entry stays the
tail append.

用 prepare_perf_changelog_merge resolve-conflict 解决与 #4050 的
perf-changelog 冲突,本 PR 的条目仍是末尾追加。

[skip-sweep]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only conflict is perf-changelog.yaml: main's file with this PR's entry
appended at the tail (prepare_perf_changelog_merge resolve-conflict).

合并 main。唯一的冲突是 perf-changelog.yaml:采用 main 的文件,并把本 PR 的条目
追加在末尾(prepare_perf_changelog_merge resolve-conflict)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@functionstackx functionstackx removed the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 8, 2026
@functionstackx functionstackx changed the title feat(agentx): add GLM-5.2 MI355X ATOM PD disaggregation points / feat(agentx): 新增 GLM-5.2 MI355X ATOM PD 分离式测试点 [Waiting for Mooncake ROCm to use madvise for THP requests for the Pollara NIC just like how AWS engineers solved it on their EFA NIC] ATOM GLM5.2 PD Nov 8, 2026
@functionstackx functionstackx changed the title [Waiting for Mooncake ROCm to use madvise for THP requests for the Pollara NIC just like how AWS engineers solved it on their EFA NIC] ATOM GLM5.2 PD [Waiting for Mooncake ROCm to use madvise for THP requests for the Pollara NIC just like how AWS engineers solved it on Mooncake EFA on Nitro EFA NIC] ATOM GLM5.2 PD Nov 8, 2026
@functionstackx functionstackx removed AMD agentx AgentX benchmarks, recipes, and infrastructure labels Nov 8, 2026
Jasen2201 and others added 3 commits November 11, 2026 04:26
…ages / Mooncake Store owner 改用 THP,不再预留 HugeTLB 页

rocm/atom-dev:nightly_202611110332 builds Mooncake 206d634, which includes
kvcache-ai/Mooncake#5151: an rdma Store owner maps its segment 2 MiB aligned
with MADV_HUGEPAGE on the NUMA node numa_exec binds it to. The owners therefore
drop MC_STORE_USE_HUGEPAGE, the variants drop their host_setup, and the
mi355x-amds hook loses its HEAD_NODE_HUGEPAGES_PER_NUMA block. The cluster's
default host setup still drops the page cache for kv-offloading dram before any
service starts, which leaves the free 2 MiB blocks the segments need.

amd-master moves to that image and records the ATOM (router) and Mooncake
source commits it ships.

rocm/atom-dev:nightly_202611110332 编译的是 Mooncake 206d634,其中包含
kvcache-ai/Mooncake#5151:rdma 的 Store owner 在 numa_exec 绑定的 NUMA 节点上
以 2 MiB 对齐映射 segment,并加 MADV_HUGEPAGE。因此 owner 去掉
MC_STORE_USE_HUGEPAGE,各 variant 去掉 host_setup,mi355x-amds hook 去掉
HEAD_NODE_HUGEPAGES_PER_NUMA 这段。kv-offloading 为 dram 时,集群默认的
host setup 仍会在所有服务启动前 drop page cache,留出 segment 需要的空闲 2 MiB 块。

amd-master 换成该镜像,并记录它所带的 ATOM(router)和 Mooncake 源码 commit。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jasen2201 Jasen2201 added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 11, 2026
@Jasen2201

Copy link
Copy Markdown
Collaborator Author

@functionstackx This is now fixed in Mooncake: kvcache-ai/Mooncake#5151 (merged as 206d634) makes rdma Store owners request THP with madvise(MADV_HUGEPAGE), the same way kvcache-ai/Mooncake#4393 did for EFA.

The PR now relies on it and reserves no hugepages:

  • The image is rocm/atom-dev:nightly_202611110332, which builds Mooncake at 206d634.
  • The owners no longer set MC_STORE_USE_HUGEPAGE, the recipe has no host_setup, and the PR no longer changes hooks/mi355x-amds/setup.sh.
  • Preflight 1P1D c112 (run 42959567550): both 768 GiB owners logged Allocated THP buffer and registered in about 15 s; 0 request errors.

The full re-sweep is running: run 42972857386.

@functionstackx

Copy link
Copy Markdown
Collaborator

Thanks @Jasen2201 it seems like using madvise 2MB hugepages ends up working. Waiting for sweep to pass

@Jasen2201 Jasen2201 changed the title [Waiting for Mooncake ROCm to use madvise for THP requests for the Pollara NIC just like how AWS engineers solved it on Mooncake EFA on Nitro EFA NIC] ATOM GLM5.2 PD feat(agentx): add GLM-5.2 MI355X ATOM PD disaggregation points / feat(agentx): 新增 GLM-5.2 MI355X ATOM PD 分离式测试点 Nov 11, 2026
Resolves the perf-changelog conflict only: main's new entries come first and
this PR's entry stays at the tail, unchanged. The sweep of this config,
42972857386 on 00005f5, remains the run to reuse:

- The recipe, the master config entry and the changelog entry are
  byte-identical to 00005f5, and `infx generate` gives the same 9 recipes
  and manifest, fingerprints included.
- main 68d69ac..88a44f1 reaches these jobs' launch path in two places,
  and neither changes them: runners.yaml adds a Kimi-K3.1-Pro model-cache
  entry for mi355x-amds, and infx/matrix/plan.py adds --repository /
  --pull-request filtering for push-to-main changelog metadata. The rest is
  CI-bot workflows, other models' recipes and the pretraining/torchtitan
  submodule. The srt-slurm pin and benchmark-multinode-tmpl.yml are
  unchanged.

只解决 perf-changelog 冲突:main 的新条目在前,本 PR 的条目仍在末尾且未改动。
该配置的扫描 42972857386(00005f531)仍是要复用的运行:

- 配方、master 配置条目和 changelog 条目与 00005f5 逐字节相同,
  `infx generate` 生成的 9 个配方和 manifest(含指纹)完全一致。
- main 68d69ac..88a44f1 只有两处触及这些作业的启动路径,均不改变作业:
  runners.yaml 为 mi355x-amds 新增 Kimi-K3.1-Pro 的模型缓存条目;
  infx/matrix/plan.py 为推送到 main 的 changelog 元数据新增
  --repository / --pull-request 过滤。其余改动是 CI 机器人工作流、其他模型的
  配方和 pretraining/torchtitan 子模块。srt-slurm 固定版本和
  benchmark-multinode-tmpl.yml 均未改变。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 42972857386

@seungrokj

Copy link
Copy Markdown
Collaborator

/use 42972857386

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

@seungrokj staged run 42972857386.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…e diff

- Add GLM-5.2 MXFP4 on MI355X with ATOM prefill/decode disaggregation behind AToMesh, Mooncake RDMA transfer and MTP, on rocm/atom-dev:nightly_202611110332 (ATOM 920a200a3, Mooncake 206d634), pinned by digest.
- Every point has one PP4 x TP1 prefill (layers 20/20/20/18) with a Mooncake Store prefix tier in DRAM: 1P1D with a colocated TP4 DCP4 decode at concurrency 56, 72, 96, 112 and 128; 1P2D with two TP4 DCP4 decodes at 56; 1P3D with three TP4 decodes at 12, 16 and 24.
- MTP draft length 3 for 1P1D, 4 for 1P2D and 5 for 1P3D. 1P1D shares one Store pool (2 x 768 GiB owners); 1P2D and 1P3D use a pool per prefill NIC (4 x 349 GiB) and cache-aware routing. dram-utilization is 1.0.
- Mooncake 206d634 includes kvcache-ai/Mooncake#5151: the RDMA Store owners back their segments with THP through madvise, so no hugepages are reserved.
- Weights run as shipped in amd/GLM-5.2-MXFP4: MXFP4 routed and shared experts in layers 3-77, BF16 elsewhere, with no online quantization flag. ATOM's default MLA absorb runs the kv_b_proj up-projections as FP8 batched GEMMs. All-reduce is BF16 (AITER quick reduce unset, default NONE) and the KV and indexer caches are FP8.
- No change to the MTP draft (model.layers.78): it is stored in BF16, including its routed and shared experts, and is served through the image defaults with no draft override, so its kv_b_proj takes the same FP8 MLA absorb as the target and it shares the target's FP8 KV and indexer caches.

- 在 MI355X 上新增 GLM-5.2 MXFP4 的 ATOM prefill/decode 分离部署,使用 AToMesh 路由、Mooncake RDMA 传输与 MTP,镜像为 rocm/atom-dev:nightly_202611110332(ATOM 920a200a3、Mooncake 206d634),按 digest 固定。
- 每个点都有一个 PP4 x TP1 prefill(各级层数 20/20/20/18),并在 DRAM 中使用 Mooncake Store 前缀缓存层:1P1D 为同机 TP4 DCP4 decode,并发 56、72、96、112、128;1P2D 为两个 TP4 DCP4 decode,并发 56;1P3D 为三个 TP4 decode,并发 12、16、24。
- MTP 草稿长度:1P1D 为 3,1P2D 为 4,1P3D 为 5。1P1D 共享一个 Store 池(2 x 768 GiB owner);1P2D 与 1P3D 每个 prefill 网卡一个池(4 x 349 GiB),并使用缓存感知路由。dram-utilization 为 1.0。
- Mooncake 206d634 包含 kvcache-ai/Mooncake#5151:RDMA Store owner 通过 madvise 以 THP 承载其内存段,因此不预留 hugepages。
- 权重按 amd/GLM-5.2-MXFP4 原样运行:第 3-77 层的路由专家与共享专家为 MXFP4,其余为 BF16,未设置在线量化参数。ATOM 默认的 MLA absorb 将 kv_b_proj 上投影以 FP8 批量 GEMM 运行。All-reduce 为 BF16(未设置 AITER quick reduce,默认 NONE),KV 缓存与 indexer 缓存为 FP8。
- MTP 草稿(model.layers.78)无变化:以 BF16 存储(包括其路由专家与共享专家),通过镜像默认设置服务、无草稿覆盖,因此其 kv_b_proj 与目标模型一样走默认的 FP8 MLA absorb,并共享目标模型的 FP8 KV 与 indexer 缓存。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj

Copy link
Copy Markdown
Collaborator

checked again and edited #4045 (comment)

Co-authored-by: Perplexity <noreply@perplexity.ai>
@edwingao28
edwingao28 dismissed stale reviews from functionstackx and cquil11 November 11, 2026 20:52

Fixed

@edwingao28
edwingao28 merged commit cd60c1e into main Nov 11, 2026
@edwingao28
edwingao28 deleted the jasen/glm52-mi355x-atom-disagg-agentx branch November 11, 2026 20:53
@github-actions

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 9 0 9 0 0
eval 4 4 0 0 0
trace 9 0 0 0 0
power 0 0 9 0 0

benchmarkFiles 10, rawBenchmarkRows 18, coveredByPerJobFiles 9, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 0, powerPublicationPoints 9, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

@functionstackx functionstackx removed the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants