Skip to content

Commit a5db3a3

Browse files
committed
feat(agentx): run GLM-5.2 FP4 MI355X ATOM TP8 C1/C2 on mono decode
Track the TP8 recipe that ROCm/ATOM PR 2736 adds to recipes/Agentic-GLM-5.2.md, on rocm/atom-dev:nightly_202610271459 (ATOM 06dbf6692), the first nightly that contains it. - TP8 C1/C2 run the GLM-5.2 mono decode (ATOM_MONO_ENABLE=1) at 3 draft tokens (AL 2.99), max-num-seqs equal to CONC, graphs [1] and [1,2]: the TP4 C1/C2 command with -tp 8. TP8 C4 and every TP4/DCP4 point keep their settings and rerun only for the image. - The atomesh router digest moves to the same nightly (5332ae78). 中文:按 ROCm/ATOM PR 2736 加入 recipes/Agentic-GLM-5.2.md 的 TP8 配方更新 MI355X ATOM AgentX 配置,镜像换为首个包含该 PR 的 rocm/atom-dev:nightly_202610271459(ATOM 06dbf6692)。TP8 C1/C2 使用 GLM-5.2 mono decode(ATOM_MONO_ENABLE=1),3 个 draft token(AL 2.99), max-num-seqs 等于并发数,graph 为 [1] 和 [1,2],即 TP4 C1/C2 命令改为 -tp 8。TP8 C4 及所有 TP4/DCP4 点位设置不变,只因换镜像而重跑。atomesh router digest 同步为同一 nightly(5332ae78)。 Signed-off-by: zhuyuhua-v <yuhzhu@amd.com>
1 parent d1cb286 commit a5db3a3

3 files changed

Lines changed: 28 additions & 12 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ base:
66
gpus_per_node: 8
77
frontend:
88
type: atomesh
9-
container_image: rocm/atom-dev@sha256:1bb1909ada9e3d178e315a3f89d041c05c7e90e7cc32de8d507ff5ab4cd4702c
9+
container_image: rocm/atom-dev@sha256:5332ae78c51985d1bd8462935bdd80817f12350c39f8e8db7d93ab9037ff5f1c
1010
enable_multiple_frontends: false
1111
observability:
1212
enabled: false
@@ -33,15 +33,15 @@ base:
3333
PYTHONNOUSERSITE: '1'
3434
AITER_QUICK_REDUCE_QUANTIZATION: INT4
3535
AITER_USE_FLYDSL_MOE_SORTING: '1'
36-
# GLM-5.2 mono decode is on by default in ATOM; only TP4 C1/C2 run it.
36+
# GLM-5.2 mono decode is on by default in ATOM; only TP4/TP8 C1/C2 run it.
3737
ATOM_MONO_ENABLE: '0'
3838
benchmark:
3939
env:
4040
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'atom:'
4141
AIPERF_APPLY_CHAT_TEMPLATE: 'true'
4242

43-
# One variant per point. Admission is 2x CONC, except TP4 C1/C2, which admit
44-
# exactly CONC with graphs captured up to it. TP4 C1/C2 run mono decode at
43+
# One variant per point. Admission is 2x CONC, except TP4/TP8 C1/C2, which admit
44+
# exactly CONC with graphs captured up to it. TP4/TP8 C1/C2 run mono decode at
4545
# three draft tokens; C4/C8 draft five and C10 four on the production path. The
4646
# DCP4 arm shards the decode KV, adds the LMCache DRAM tier at 256 GiB per rank
4747
# (1024 GiB, 1099.5 GB per server, within the 1109 GB TP4 share at
@@ -210,9 +210,11 @@ override_tp8_c1:
210210
agg:
211211
gpus: 8
212212
args:
213-
max-num-seqs: 2
214-
cudagraph-capture-sizes: '[1,2]'
215-
num-speculative-tokens: 5
213+
max-num-seqs: 1
214+
cudagraph-capture-sizes: '[1]'
215+
num-speculative-tokens: 3
216+
env:
217+
ATOM_MONO_ENABLE: '1'
216218
benchmark:
217219
env:
218220
CONC: '1'
@@ -222,9 +224,11 @@ override_tp8_c2:
222224
agg:
223225
gpus: 8
224226
args:
225-
max-num-seqs: 4
226-
cudagraph-capture-sizes: '[1,2,4]'
227-
num-speculative-tokens: 5
227+
max-num-seqs: 2
228+
cudagraph-capture-sizes: '[1,2]'
229+
num-speculative-tokens: 3
230+
env:
231+
ATOM_MONO_ENABLE: '1'
228232
benchmark:
229233
env:
230234
CONC: '2'

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1495,9 +1495,10 @@ glm5.2-fp8-mi355x-sglang-agentic-mtp:
14951495
# C1/C2 run the GLM-5.2 mono decode at K3. Large concurrency (C16-C48) adds DCP4
14961496
# to shard the decode KV and the LMCache DRAM tier. The recipe sets the draft
14971497
# depth per point (K3 for C1/C2 and C48, K5 for C4/C8, K4 for C10-C40). The TP8
1498-
# arm stays GPU-resident with no KV offload.
1498+
# arm stays GPU-resident with no KV offload; TP8 C1/C2 also run mono decode at
1499+
# K3 (ROCm/ATOM PR 2736) and C4 stays at K5.
14991500
glm5.2-fp4-mi355x-atom-agentic-mtp:
1500-
image: rocm/atom-dev:nightly_202610181506
1501+
image: rocm/atom-dev:nightly_202610271459
15011502
model: amd/GLM-5.2-MXFP4
15021503
model-prefix: glm5.2
15031504
runner: cluster:mi355x-amds

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11556,3 +11556,14 @@
1155611556
- "Disagg: 6 DEP2 prefill workers with a DEP8 decode at conc 280-360 or a DEP16 decode at conc 300; DeepGEMM MegaMoE, NIXL transfer plus a shared MooncakeStore DRAM pool (160 GB per rank), Dynamo KV router."
1155711557
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4468
1155811558
scenario-type: [agentic-coding]
11559+
11560+
- config-keys:
11561+
- glm5.2-fp4-mi355x-atom-agentic-mtp
11562+
scenario-type:
11563+
- agentic-coding
11564+
description:
11565+
- "Image rocm/atom-dev:nightly_202610181506 (ATOM 0.1.8.dev77+gdf86c4829) -> nightly_202610271459 (ATOM 0.1.9.dev59+g06dbf6692, digest sha256:5332ae78c51985d1bd8462935bdd80817f12350c39f8e8db7d93ab9037ff5f1c), the first nightly that contains ROCm/ATOM PR 2736. The atomesh router container_image digest moves from 1bb1909a to 5332ae78, the same nightly as the workers."
11566+
- "TP8 C1 and C2 follow the TP8 recipe added to recipes/Agentic-GLM-5.2.md by ROCm/ATOM PR 2736: the TP4 C1/C2 command with -tp 8. Both run the GLM-5.2 mono decode (ATOM_MONO_ENABLE=1) on the external-indexer path at 3 draft tokens (golden AL 2.99), with max-num-seqs equal to the concurrency and CUDA graphs at [1] and [1,2]. C1 was 5 draft tokens, max-num-seqs 2 and graphs [1,2]; C2 was 5 draft tokens, max-num-seqs 4 and graphs [1,2,4]."
11567+
- "TP8 C4, TP4 C1-C10 and TP4+DCP4 C16-C48 keep their settings and ATOM_MONO_ENABLE values; they rerun only for the image change. LMCache stays at 0.5.6.dev139+gf22dec28, the wheel pinned in both images."
11568+
- "ATOM changes in the new image that matter here: mono decode admits TP8 and keeps the fused indexer at TP4 only (ROCm/ATOM #2736). Shared paths the TP4, DCP4 and TP8 C4 points also pick up: one graph mechanism per region, including the glm5_next indexer and kpool (#2688); MoE routing before the rows move (#2725); chunked DSA indexer decode logits (#2607); speculative output buffer reuse (#2670); MLA Triton decode on the maintained AITER API (#2669)."
11569+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4608

0 commit comments

Comments
 (0)