Skip to content

feat(agentx): GLM-5.2 FP4 MI355X ATOM on the Quark MXFP4 AttnFP8 checkpoint, TP8 C1/C2 on mono decode / feat(agentx)：GLM-5.2 FP4 MI355X ATOM 换用 Quark MXFP4 AttnFP8 checkpoint，TP8 C1/C2 改用 mono decode #4608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zhuyuhua-v wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat(agentx): GLM-5.2 FP4 MI355X ATOM on the Quark MXFP4 AttnFP8 checkpoint, TP8 C1/C2 on mono decode / feat(agentx)：GLM-5.2 FP4 MI355X ATOM 换用 Quark MXFP4 AttnFP8 checkpoint，TP8 C1/C2 改用 mono decode #4608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a5db3a3
feat(agentx): run GLM-5.2 FP4 MI355X ATOM TP8 C1/C2 on mono decode
zhuyuhua-v Oct 29, 2026
b202405
Merge branch 'main' into amd/glm5.2-fp4-mi355x-atom-tp8-mono-1027
seungrokj Nov 2, 2026
f17f881
Merge remote-tracking branch 'origin/main' into amd/glm5.2-fp4-mi355x…
zhuyuhua-v Nov 4, 2026
1e7bb50
feat(agentx): serve GLM-5.2 MI355X ATOM on the Quark MXFP4 AttnFP8 ch…
zhuyuhua-v Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions ...ncex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ base:
gpus_per_node: 8
frontend:
type: atomesh
container_image: rocm/atom-dev@sha256:1bb1909ada9e3d178e315a3f89d041c05c7e90e7cc32de8d507ff5ab4cd4702c
container_image: rocm/atom-dev@sha256:5332ae78c51985d1bd8462935bdd80817f12350c39f8e8db7d93ab9037ff5f1c
enable_multiple_frontends: false
observability:
enabled: false
Expand All @@ -18,30 +18,26 @@ base:
nodes: 1
workers: 1
args:
served-model-name: amd/GLM-5.2-MXFP4
served-model-name: amd/GLM-5.2-Quark-MXFP4-AttnFP8
gpu-memory-utilization: 0.95
enable_prefix_caching: true
# model.layers.78 is the MTP head, shipped in BF16; excluding it keeps
# online ptpc_fp8 quantization off the draft while experts stay MXFP4.
online_quant_config: '{"global_quant_config":"ptpc_fp8","exclude_layer":["lm_head","model.embed_tokens","*.mlp.gate","model.layers.[0-9].mlp.*expert*","model.layers.[1-6][0-9].mlp.*expert*","model.layers.7[0-7].mlp.*expert*","model.layers.78.*"]}'
max-num-batched-tokens: 16384
kv_cache_dtype: fp8
index_cache_dtype: fp4
block-size: 64
method: mtp
env:
PYTHONNOUSERSITE: '1'
AITER_QUICK_REDUCE_QUANTIZATION: INT4
AITER_USE_FLYDSL_MOE_SORTING: '1'
# GLM-5.2 mono decode is on by default in ATOM; only TP4 C1/C2 run it.
# GLM-5.2 mono decode is on by default in ATOM; only TP4/TP8 C1/C2 run it.
ATOM_MONO_ENABLE: '0'
benchmark:
env:
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'atom:'
AIPERF_APPLY_CHAT_TEMPLATE: 'true'

# One variant per point. Admission is 2x CONC, except TP4 C1/C2, which admit
# exactly CONC with graphs captured up to it. TP4 C1/C2 run mono decode at
# One variant per point. Admission is 2x CONC, except TP4/TP8 C1/C2, which admit
# exactly CONC with graphs captured up to it. TP4/TP8 C1/C2 run mono decode at
# three draft tokens; C4/C8 draft five and C10 four on the production path. The
# DCP4 arm shards the decode KV, adds the LMCache DRAM tier at 256 GiB per rank
# (1024 GiB, 1099.5 GB per server, within the 1109 GB TP4 share at
Expand Down Expand Up @@ -210,9 +206,11 @@ override_tp8_c1:
agg:
gpus: 8
args:
max-num-seqs: 2
cudagraph-capture-sizes: '[1,2]'
num-speculative-tokens: 5
max-num-seqs: 1
cudagraph-capture-sizes: '[1]'
num-speculative-tokens: 3
env:
ATOM_MONO_ENABLE: '1'
benchmark:
env:
CONC: '1'
Expand All @@ -222,9 +220,11 @@ override_tp8_c2:
agg:
gpus: 8
args:
max-num-seqs: 4
cudagraph-capture-sizes: '[1,2,4]'
num-speculative-tokens: 5
max-num-seqs: 2
cudagraph-capture-sizes: '[1,2]'
num-speculative-tokens: 3
env:
ATOM_MONO_ENABLE: '1'
Comment on lines 220 to +227

@claude claude Bot Oct 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The newly enabled TP8 mono-decode path gets no GSM8K accuracy check, so a TP8-only correctness regression would publish unflagged. Eval rows are grouped by engine env alone, and TP8 C1/C2 now carry the same env as TP4 C1/C2. The group picks one row at the highest concurrency, and the tie at C2 goes to TP4 C2, listed first. Fix: give TP8 mono its own eval, for example a tp: 8 C1/C2 row in the -mono key with grouping that keeps TP8 apart from TP4.

Why this was flagged

override_tp8_c1/override_tp8_c2 (agentic.yaml:216-217, 230-231) set ATOM_MONO_ENABLE: '1', which gives them the same engine environment as override_tp4_c1/c2. fingerprint.py:81 engine_environment uses only role env. generate.py:802-812 groups single-node agentic rows by model/runner/framework/precision/spec/dp-attn/image plus _engine_env, with no tp field. generate.py:815 then picks max(entries, key=conc). Search-space order at amd-master.yaml:1516-1518 lists TP4 first, so TP4 C2 wins the C2 tie. Per the PR, ATOM keeps the fused indexer at TP4 only, so TP8 mono runs a different external-indexer kernel path. The PR also says TP8 C2 mono has never been measured on GPU. Both the main key and the -mono key (TP4 only, amd-master.yaml:1538) evaluate only TP4 C2. As a result, TP8 C1/C2 throughput from a code path with no accuracy check gets published to the dashboard. On base, TP8 C1/C2 ran the production non-mono path, which the TP4 C10 eval in the same env group covered.

Verification: agentic.yaml:216-217 and 230-231 now set ATOM_MONO_ENABLE: '1' for override_tp8_c1/c2, identical to override_tp4_c1/c2. generate.py:802-811 groups by _engine_env with no TP field, and generate.py:815 picks max(entries, key=conc), so the C2 tie goes to TP4 C2. TP8 C1/C2 therefore never get a GSM8K run.

benchmark:
env:
CONC: '2'
Expand Down
7 changes: 4 additions & 3 deletions inferencex-e2e/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -848,10 +848,11 @@ glm5.2-fp8-mi355x-sglang-agentic-mtp:
# C1/C2 run the GLM-5.2 mono decode at K3. Large concurrency (C16-C48) adds DCP4
# to shard the decode KV and the LMCache DRAM tier. The recipe sets the draft
# depth per point (K3 for C1/C2 and C48, K5 for C4/C8, K4 for C10-C40). The TP8
# arm stays GPU-resident with no KV offload.
# arm stays GPU-resident with no KV offload; TP8 C1/C2 also run mono decode at
# K3 (ROCm/ATOM PR 2736) and C4 stays at K5.
Comment on lines +851 to +852

@claude claude Bot Oct 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): The added comment text (ROCm/ATOM PR 2736) records where the setting came from, which AGENTS.md's no-history comment rule forbids. Fix: drop the PR reference and keep only the technical note (TP8 C1/C2 run mono decode at K3, C4 stays at K5). The changelog entry already records the provenance.

Why this was flagged

AGENTS.md:57 says comments must not record history such as what a value replaced or where it came from, because git and perf-changelog.yaml are the record. The diff adds K3 (ROCm/ATOM PR 2736) and C4 stays at K5. to the amd-master.yaml comment at 1498-1499. That provenance already appears in the new perf-changelog.yaml block.

Verification: Triggered whenever this comment merges as written. AGENTS.md:57 says "No history: how code evolved, what it replaced, where it came from, or past results." The parenthetical "(ROCm/ATOM PR 2736)" at amd-master.yaml lines 1498-1499 cites the upstream PR that introduced the setting, which is provenance. Nothing breaks at runtime.

glm5.2-fp4-mi355x-atom-agentic-mtp:
image: rocm/atom-dev:nightly_202610181506
model: amd/GLM-5.2-MXFP4
image: rocm/atom-dev:nightly_202610271459
model: amd/GLM-5.2-Quark-MXFP4-AttnFP8
model-prefix: glm5.2
runner: cluster:mi355x-amds
Comment on lines +851 to 857

@claude claude Bot Oct 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The -mono eval key now runs old-nightly workers behind a new-nightly router, and the changelog never lists it. The glm5.2-fp4-mi355x-atom-agentic-mtp-mono key (amd-master.yaml:1525) still uses nightly_202610181506. It shares srt-recipe-dir: glm5.2/atom/mi355x-fp4-mtp, whose atomesh router digest moved to 5332ae78. Fix: bump the -mono image to nightly_202610271459 and add it to the new block's config-keys. That keeps router and workers on one nightly for both keys, which infx.matrix.changed already flags.

Why this was flagged

The recipe's container_image in agentic.yaml:9 moved from the old nightly's digest 1bb1909a to 5332ae78, which the PR says is nightly_202610271459. glm5.2-fp4-mi355x-atom-agentic-mtp-mono at amd-master.yaml:1525-1526 keeps image: rocm/atom-dev:nightly_202610181506 and reads the same recipe. Its GSM8K eval therefore pairs an ATOM 0.1.8 worker with an ATOM 0.1.9 router. On the base branch both were the same nightly. The PR description says infx.matrix.changed lists the -mono key. AGENTS.md:87 requires a changelog entry for the affected keys, but perf-changelog.yaml's new block lists only glm5.2-fp4-mi355x-atom-agentic-mtp, so no results are collected for the changed -mono run.

Verification: agentic.yaml:9 moves container_image: rocm/atom-dev@ sha256:1bb1909a... to @ sha256:5332ae78.... The -mono key at amd-master.yaml:1525-1526 keeps image: rocm/atom-dev:nightly_202610181506. After this change, -mono runs an old-nightly worker behind a new-nightly router. -mono appears nowhere in perf-changelog.yaml. validate_perf_changelog.py does not enforce coverage.

precision: fp4
Expand Down
14 changes: 14 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11989,3 +11989,17 @@
- "Update the Muse Spark Open GB300 curve to lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc and retune request admission, prefill-decode interval, scheduling and host cache per concurrency."
- "Add TP4/EP4 concurrency 4 and 40; drop concurrency 5, 28 and 34."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4724

- config-keys:
- glm5.2-fp4-mi355x-atom-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Image rocm/atom-dev:nightly_202610181506 (ATOM 0.1.8.dev77+gdf86c4829) -> nightly_202610271459 (ATOM 0.1.9.dev59+g06dbf6692, digest sha256:5332ae78c51985d1bd8462935bdd80817f12350c39f8e8db7d93ab9037ff5f1c), the first nightly that contains ROCm/ATOM PR 2736. The atomesh router container_image digest moves from 1bb1909a to 5332ae78, the same nightly as the workers."
- "Checkpoint amd/GLM-5.2-MXFP4 -> amd/GLM-5.2-Quark-MXFP4-AttnFP8 (Hugging Face revision 985ea5ad), served as shipped with no online_quant_config. It stores attention q_a/q_b/kv_a/kv_b/o_proj and the indexer wq_b/wk in PTPC FP8 (static per-channel weights, dynamic per-token activations) and every other linear in MXFP4, including the routed and shared experts, the layer 0-2 dense MLP and the MTP layer 78; lm_head, eh_proj, the MoE routers and indexer.weights_proj stay BF16. The previous recipe quantized the BF16 attention, indexer and dense MLP of amd/GLM-5.2-MXFP4 to PTPC FP8 at load time through online_quant_config."
- "Remove AITER_QUICK_REDUCE_QUANTIZATION=INT4, so tensor-parallel all-reduce is no longer integer-quantized. In the pinned AITER, INT4 quick reduce took BF16 messages of at least 16 MiB at TP4 (about 1,366 tokens of the 6144-wide hidden state, so prefill steps only) and at least 2 GiB at TP8, which no GLM-5.2 step reaches."
- "Draft: the MTP head is the checkpoint's own layer 78 at its stored precision (MXFP4 experts and MLP, PTPC FP8 attention, BF16 eh_proj), with no draft-related quantization setting in the recipe. It was BF16 in amd/GLM-5.2-MXFP4."
- "TP8 C1 and C2 follow the TP8 recipe added to recipes/Agentic-GLM-5.2.md by ROCm/ATOM PR 2736: the TP4 C1/C2 command with -tp 8. Both run the GLM-5.2 mono decode (ATOM_MONO_ENABLE=1) on the external-indexer path at 3 draft tokens (golden AL 2.99), with max-num-seqs equal to the concurrency and CUDA graphs at [1] and [1,2]. C1 was 5 draft tokens, max-num-seqs 2 and graphs [1,2]; C2 was 5 draft tokens, max-num-seqs 4 and graphs [1,2,4]."
- "TP8 C4, TP4 C1-C10 and TP4+DCP4 C16-C48 keep their settings and ATOM_MONO_ENABLE values; they rerun for the image and checkpoint changes. LMCache stays at 0.5.6.dev139+gf22dec28, the wheel pinned in both images."
- "ATOM changes in the new image that matter here: mono decode admits TP8 and keeps the fused indexer at TP4 only (ROCm/ATOM #2736). Shared paths the TP4, DCP4 and TP8 C4 points also pick up: one graph mechanism per region, including the glm5_next indexer and kpool (#2688); MoE routing before the rows move (#2725); chunked DSA indexer decode logits (#2607); speculative output buffer reuse (#2670); MLA Triton decode on the maintained AITER API (#2669)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4608
Loading