Repository navigation
feat(agentx): run GLM-5.2 FP4 MI355X ATOM TP8 C1/C2 on mono decode / feat(agentx)：GLM-5.2 FP4 MI355X ATOM TP8 C1/C2 改用 mono decode - #4608
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
6b94f18 to
cf84a73
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # arm stays GPU-resident with no KV offload; TP8 C1/C2 also run mono decode at | ||
| # K3 (ROCm/ATOM PR 2736) and C4 stays at K5. | ||
| glm5.2-fp4-mi355x-atom-agentic-mtp: | ||
| image: rocm/atom-dev:nightly_202610181506 | ||
| image: rocm/atom-dev:nightly_202610271459 | ||
| model: amd/GLM-5.2-MXFP4 | ||
| model-prefix: glm5.2 | ||
| runner: cluster:mi355x-amds |
There was a problem hiding this comment.
🟡 (optional) The -mono eval key now runs old-nightly workers behind a new-nightly router, and the changelog never lists it. The glm5.2-fp4-mi355x-atom-agentic-mtp-mono key (amd-master.yaml:1525) still uses nightly_202610181506. It shares srt-recipe-dir: glm5.2/atom/mi355x-fp4-mtp, whose atomesh router digest moved to 5332ae78. Fix: bump the -mono image to nightly_202610271459 and add it to the new block's config-keys. That keeps router and workers on one nightly for both keys, which infx.matrix.changed already flags.
Why this was flagged
The recipe's container_image in agentic.yaml:9 moved from the old nightly's digest 1bb1909a to 5332ae78, which the PR says is nightly_202610271459. glm5.2-fp4-mi355x-atom-agentic-mtp-mono at amd-master.yaml:1525-1526 keeps image: rocm/atom-dev:nightly_202610181506 and reads the same recipe. Its GSM8K eval therefore pairs an ATOM 0.1.8 worker with an ATOM 0.1.9 router. On the base branch both were the same nightly. The PR description says infx.matrix.changed lists the -mono key. AGENTS.md:87 requires a changelog entry for the affected keys, but perf-changelog.yaml's new block lists only glm5.2-fp4-mi355x-atom-agentic-mtp, so no results are collected for the changed -mono run.
Verification: agentic.yaml:9 moves container_image: rocm/atom-dev@ sha256:1bb1909a... to @ sha256:5332ae78.... The -mono key at amd-master.yaml:1525-1526 keeps image: rocm/atom-dev:nightly_202610181506. After this change, -mono runs an old-nightly worker behind a new-nightly router. -mono appears nowhere in perf-changelog.yaml. validate_perf_changelog.py does not enforce coverage.
| agg: | ||
| gpus: 8 | ||
| args: | ||
| max-num-seqs: 4 | ||
| cudagraph-capture-sizes: '[1,2,4]' | ||
| num-speculative-tokens: 5 | ||
| max-num-seqs: 2 | ||
| cudagraph-capture-sizes: '[1,2]' | ||
| num-speculative-tokens: 3 | ||
| env: | ||
| ATOM_MONO_ENABLE: '1' |
There was a problem hiding this comment.
🟡 (optional) The newly enabled TP8 mono-decode path gets no GSM8K accuracy check, so a TP8-only correctness regression would publish unflagged. Eval rows are grouped by engine env alone, and TP8 C1/C2 now carry the same env as TP4 C1/C2. The group picks one row at the highest concurrency, and the tie at C2 goes to TP4 C2, listed first. Fix: give TP8 mono its own eval, for example a tp: 8 C1/C2 row in the -mono key with grouping that keeps TP8 apart from TP4.
Why this was flagged
override_tp8_c1/override_tp8_c2 (agentic.yaml:216-217, 230-231) set ATOM_MONO_ENABLE: '1', which gives them the same engine environment as override_tp4_c1/c2. fingerprint.py:81 engine_environment uses only role env. generate.py:802-812 groups single-node agentic rows by model/runner/framework/precision/spec/dp-attn/image plus _engine_env, with no tp field. generate.py:815 then picks max(entries, key=conc). Search-space order at amd-master.yaml:1516-1518 lists TP4 first, so TP4 C2 wins the C2 tie. Per the PR, ATOM keeps the fused indexer at TP4 only, so TP8 mono runs a different external-indexer kernel path. The PR also says TP8 C2 mono has never been measured on GPU. Both the main key and the -mono key (TP4 only, amd-master.yaml:1538) evaluate only TP4 C2. As a result, TP8 C1/C2 throughput from a code path with no accuracy check gets published to the dashboard. On base, TP8 C1/C2 ran the production non-mono path, which the TP4 C10 eval in the same env group covered.
Verification: agentic.yaml:216-217 and 230-231 now set ATOM_MONO_ENABLE: '1' for override_tp8_c1/c2, identical to override_tp4_c1/c2. generate.py:802-811 groups by _engine_env with no TP field, and generate.py:815 picks max(entries, key=conc), so the C2 tie goes to TP4 C2. TP8 C1/C2 therefore never get a GSM8K run.
| # arm stays GPU-resident with no KV offload; TP8 C1/C2 also run mono decode at | ||
| # K3 (ROCm/ATOM PR 2736) and C4 stays at K5. |
There was a problem hiding this comment.
🟡 nit (optional): The added comment text (ROCm/ATOM PR 2736) records where the setting came from, which AGENTS.md's no-history comment rule forbids. Fix: drop the PR reference and keep only the technical note (TP8 C1/C2 run mono decode at K3, C4 stays at K5). The changelog entry already records the provenance.
Why this was flagged
AGENTS.md:57 says comments must not record history such as what a value replaced or where it came from, because git and perf-changelog.yaml are the record. The diff adds K3 (ROCm/ATOM PR 2736) and C4 stays at K5. to the amd-master.yaml comment at 1498-1499. That provenance already appears in the new perf-changelog.yaml block.
Verification: Triggered whenever this comment merges as written. AGENTS.md:57 says "No history: how code evolved, what it replaced, where it came from, or past results." The parenthetical "(ROCm/ATOM PR 2736)" at amd-master.yaml lines 1498-1499 cites the upstream PR that introduced the setting, which is provenance. Nothing breaks at runtime.
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
inferencex-e2e/uv-0.13.6-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:1— A 17 MB uv wheel is committed to the project root. Every clone and checkout now carries this unused binary permanently in git history. Nothing in the repo referencesuv-0.13.6, and it has nothing to do with the GLM-5.2 config change. AGENTS.md routes dependency changes through pyproject.toml and uv.lock. Fix: remove the file from the commit, for example withgit rm --cachedand an amended or squashed commit, so it never reaches main.Why this was flagged
The commit adds inferencex-e2e/uv-0.13.6-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17333810 bytes). A repo-wide grep for
uv-0.13.6finds nothing, so no build, CI or pyproject file uses it. After merge, every clone, CI checkout and runner sync downloads the extra 17 MB, and squash-merging keeps it in main's history for good. The base branch has no such artifact. No .gitignore rule or other check blocked it.Verification: The trigger is merging the PR as it stands.
git diff --stat 6cbbb96e..HEADshows the wheel as a new file:Bin 0 -> 17333810 bytes, and the base branch does not have this file. A repo-wide grep foruv-0\.13\.6|uv==0\.13finds no files. The root .gitignore has no*.whlrule. A vendored uv wheel at the project root breaks AGENTS.md line 82.
There was a problem hiding this comment.
🟡 Changes recommended
The shared router-image update changes the mono configuration’s worker/router combination, but its omission from the changelog leaves it without required sweep or evaluation coverage.
1 open issue
What changed in this PR
Updates the GLM-5.2 FP4 MI355X ATOM AgentX configuration to enable mono decode for TP8 at concurrency 1 and 2, using a newer ATOM image with TP8 support.
Changes:
- Advances the main configuration’s worker image and shared router digest.
- Enables TP8 C1/C2 mono decode with three draft tokens and concurrency-matched admission and graph sizes.
- Adds a performance changelog entry to rerun the main configuration; other points retain their tuning.
| File | Description |
|---|---|
inferencex-e2e/perf-changelog.yaml |
Records the image upgrade, TP8 tuning, and main-key sweep scope. |
inferencex-e2e/configs/amd-master.yaml |
Updates the main worker image and explanatory comments. |
inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.2/atom/mi355x-fp4-mtp/agentic.yaml |
Updates the router digest and enables mono decode for TP8 C1/C2. |
Review effort: Balanced
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4571 | ||
|
|
||
| - config-keys: | ||
| - glm5.2-fp4-mi355x-atom-agentic-mtp |
zhuyuhua-v
added
AMD
full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40876229659 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40876229659 |
Track the TP8 recipe that ROCm/ATOM PR 2736 adds to recipes/Agentic-GLM-5.2.md, on rocm/atom-dev:nightly_202610271459 (ATOM 06dbf6692), the first nightly that contains it. - TP8 C1/C2 run the GLM-5.2 mono decode (ATOM_MONO_ENABLE=1) at 3 draft tokens (AL 2.99), max-num-seqs equal to CONC, graphs [1] and [1,2]: the TP4 C1/C2 command with -tp 8. TP8 C4 and every TP4/DCP4 point keep their settings and rerun only for the image. - The atomesh router digest moves to the same nightly (5332ae78). 中文：按 ROCm/ATOM PR 2736 加入 recipes/Agentic-GLM-5.2.md 的 TP8 配方更新 MI355X ATOM AgentX 配置，镜像换为首个包含该 PR 的 rocm/atom-dev:nightly_202610271459（ATOM 06dbf6692）。TP8 C1/C2 使用 GLM-5.2 mono decode（ATOM_MONO_ENABLE=1），3 个 draft token（AL 2.99）， max-num-seqs 等于并发数，graph 为 [1] 和 [1,2]，即 TP4 C1/C2 命令改为 -tp 8。TP8 C4 及所有 TP4/DCP4 点位设置不变，只因换镜像而重跑。atomesh router digest 同步为同一 nightly（5332ae78）。 Signed-off-by: zhuyuhua-v <yuhzhu@amd.com>
cf84a73 to
a5db3a3
Compare
|
/reuse-sweep-run 40661036447 |
|
中文
|
| @@ -33,15 +33,15 @@ base: | |||
| PYTHONNOUSERSITE: '1' | |||
| AITER_QUICK_REDUCE_QUANTIZATION: INT4 | |||
There was a problem hiding this comment.
does this pass deepswe evals or BEAM on vs off? @seungrokj
There was a problem hiding this comment.
pretty sure it doesnt espcially at TP4 and TP8 lol
| @@ -33,15 +33,15 @@ base: | |||
| PYTHONNOUSERSITE: '1' | |||
| AITER_QUICK_REDUCE_QUANTIZATION: INT4 | |||
There was a problem hiding this comment.
pretty sure it doesnt espcially at TP4 and TP8 lol
Summary
ROCm/ATOM PR 2736 (merged as
f39bff7ef) opens GLM-5.2 mono decode to TP8. Before it, TP8 loggedGLM-5.2 MonoKernel off: TP 8 not in (4,). The PR's recipe runs AgentX TP8 C1/C2 as the TP4 C1/C2 command with-tp 8. This PR movesglm5.2-fp4-mi355x-atom-agentic-mtpto that recipe:rocm/atom-dev:nightly_202610181506(ATOM0.1.8.dev77+gdf86c4829) ->nightly_202610271459(ATOM0.1.9.dev59+g06dbf6692), the first nightly that containsf39bff7ef. The atomesh routercontainer_imagedigest moves from1bb1909ato5332ae78(same nightly as the workers).ATOM_MONO_ENABLE: '1') on the external-indexer path at 3 draft tokens (golden AL 2.99).max-num-seqsequals CONC, with graphs[1]/[1,2]. Before: 5 draft tokens,max-num-seqs2 / 4, graphs[1,2]/[1,2,4].ATOM_MONO_ENABLEvalues. They rerun only because the image changed. LMCache stays0.5.6.dev139+gf22dec28, the wheel both images pin.Generated matrix (
python -m infx.matrix.plan): 13 throughput jobs (TP4 C1/2/4/8/10, DCP4 C16/24/32/40/48, TP8 C1/2/4) plus the default per-engine-environment GSM8K: TP4 C2 (mono on; TP8 C1/C2 share this engine env), TP4 C10 and DCP4 C48.Validation details
Upstream GPU evidence (ROCm/ATOM PR 2736): MI355X,
nightly_202610181506= ATOMdf86c482plus the TP8 admission. These runs used the PR's patch, not the06dbf6692image.TP8 C1 accuracy: GSM8K first 250, 3-shot, real MTP3 acceptance,
num_concurrent=1.TP8 C1 performance: AgentX C1, 3,600 s, forced AL 2.99. The TP4 column is InferenceX run 39540045682, job 121211066939.
Image provenance. The digest comes from the registry
Docker-Content-Digestforrocm/atom-dev:nightly_202610271459. The image labels areorg.opencontainers.image.revision: 06dbf66920bafb22afc391a43014500c3dc704eaandorg.opencontainers.image.version: 0.1.9.dev59+g06dbf6692.git merge-base --is-ancestor f39bff7ef 06dbf6692passes. Checked the same way, the old tag resolves to1bb1909a, the digest #4308 used.Local checks:
validate_perf_changelog --base-ref origin/mainpasses.infx.matrix.changedlistsglm5.2-fp4-mi355x-atom-agentic-mtpandglm5.2-fp4-mi355x-atom-agentic-mtp-mono. The-monokey is unchanged inamd-master.yaml, but it reads the same recipe, whose router digest moved.override_tp8_c1/_c2/_c4givesmax-num-seqs1 / 2 / 8, graphs[1]/[1,2]/[1,2,4,8], drafts 3 / 3 / 5, andATOM_MONO_ENABLE1 / 1 / 0.AI model disclosure
claude-opus-5-5), via Claude Code.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
ROCm/ATOM PR 2736（已合入，
f39bff7ef）让 GLM-5.2 mono decode 支持 TP8。此前 TP8 会打印GLM-5.2 MonoKernel off: TP 8 not in (4,)。该 PR 的 recipe 中，AgentX TP8 C1/C2 就是 TP4 C1/C2 的命令改为-tp 8。本 PR 让glm5.2-fp4-mi355x-atom-agentic-mtp按这个 recipe 运行：rocm/atom-dev:nightly_202610181506（ATOM0.1.8.dev77+gdf86c4829）->nightly_202610271459（ATOM0.1.9.dev59+g06dbf6692），这是第一个包含f39bff7ef的 nightly。atomesh router 的container_imagedigest 从1bb1909a改为5332ae78，与 worker 使用同一 nightly。ATOM_MONO_ENABLE: '1'），3 个 draft token（golden AL 2.99）。max-num-seqs等于并发数，graph 为[1]/[1,2]。原配置为 5 个 draft token，max-num-seqs2 / 4，graph[1,2]/[1,2,4]。ATOM_MONO_ENABLE都不变，只因换镜像而重跑。LMCache 仍为0.5.6.dev139+gf22dec28，两个镜像固定的是同一个 wheel。生成的矩阵：13 个吞吐任务，外加按引擎环境默认选出的 GSM8K：TP4 C2（mono 开启；TP8 C1/C2 与其引擎环境相同）、TP4 C10 和 DCP4 C48。
验证数据见上方表格（ATOM PR 2736 的 TP8 C1 精度和性能，基于
df86c482加该 PR 的补丁，不是06dbf6692镜像）。本地检查：changelog 验证器通过；infx.matrix.changed列出主 key 和-monokey（后者在 amd-master.yaml 中未改，但读取同一个 recipe，其 router digest 已更新）；展开后的 TP8 recipe 参数与 ATOM recipe 一致。AI 使用说明：Claude Opus 5.5（
claude-opus-5-5，通过 Claude Code）协助修改 recipe、master config 和 changelog，运行本地检查并起草本说明；由 zhuyuhua-v 审阅并提交。