Repository navigation
perf: use Gluon for GLM-5.2 TP8 AgentX on 1111 daily / perf: GLM-5.2 TP8 AgentX 使用 Gluon 和 1111 daily 镜像 - #4748
jiejingzhangamd wants to merge 5 commits into
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
jiejingzhangamd
added
all-evals
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| - config-keys: | ||
| - glm5.2-fp4-mi355x-sglang-agentic-mtp | ||
| all-evals: true |
There was a problem hiding this comment.
🔴 After merge, the changelog run for this image bump and Gluon change produces no new throughput numbers; it schedules only evals. Setting all-evals: true on a changelog entry implies evals-only (EVALS.md:64-65). infx/matrix/plan.py:429 sets suppress_throughput whenever entry.all_evals is true. The PR says all 13 throughput points get rescheduled; the code schedules none. Fix: drop all-evals: true so the 13 points run. If the Gluon TP8 points need eval coverage on the PR run, use the all-evals PR label, which adds evals and keeps throughput (EVALS.md:66).
Why this was flagged
The new tail entry sets all-evals: true for glm5.2-fp4-mi355x-sglang-agentic-mtp (inferencex-e2e/perf-changelog.yaml:11859). In infx/matrix/plan.py:428-429, suppress_throughput = evals_only or entry.evals_only or entry.all_evals, so the if not suppress_throughput: block that builds benchmark rows is skipped for this key. Only eval rows are emitted, with mode "all". infx/tests/matrix/test_process_changelog.py:563-578 pins this: an entry with all-evals: True and no CLI flags gives mode "all" and expected_benchmarks == []. The image change (amd-master.yaml:799) and the Gluon TP8 overrides (agentic.yaml:223-224, 239-240, 255-256) therefore never get throughput results, even though the PR text says all 13 points are rescheduled. The published curve keeps the v0.5.22 numbers while the config points at v0.5.23.
Verification: In inferencex-e2e/infx/matrix/plan.py:429, suppress_throughput = evals_only or entry.evals_only or entry.all_evals, so the if not suppress_throughput: block at 431-449 never calls generate_current(..., "none", ...) for this key. Only the eval rows from 454-464 are emitted, in mode "all" (line 428/458). plan.py is not changed by this PR, so this is the current behavior.
| - "Use the SGLang ROCm 7.2.0 MI35x 20261102 daily image for GLM-5.2 MXFP4 MI355X AgentX MTP." | ||
| - "Use Gluon target MoE only for TP8/EP1 at concurrency 1, 2, and 4; preserve the checkpoint BF16 draft experts with the auto speculative MoE backend." | ||
| - "Keep TP4/EP4 HiCache and DCP4 settings unchanged and evaluate every point so the Gluon path receives accuracy coverage." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER |
There was a problem hiding this comment.
🔴 The PR's changelog check will fail, and the merge helper will refuse to rewrite the link, so this entry cannot run or merge as written. The new entry's pr-link ends in pull/PLACEHOLDER. validate_added_pr_link accepts only the exact PR URL or an entry in PR_LINK_PLACEHOLDERS, which holds only XXX and .../pull/XXX. Fix: set pr-link to the real URL https://github.com/SemiAnalysisAI/InferenceX/pull/<this PR number>, or to https://github.com/SemiAnalysisAI/InferenceX/pull/XXX so the merge tooling fills it in.
Why this was flagged
The appended block at inferencex-e2e/perf-changelog.yaml:11864 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER. On PR runs, inferencex-e2e/infx/workflows/validate_perf_changelog.py:113 raises ChangelogValidationError ("new PR entry must use ... or an XXX placeholder") for any link that is neither the expected PR URL nor in PR_LINK_PLACEHOLDERS (lines 21-24: "XXX" and "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"). On main, line 108 requires CANONICAL_PR_LINK (pull/\d+), which also rejects it. inferencex-e2e/infx/workflows/prepare_perf_changelog_merge.py:108 likewise raises "appended entry ... has unexpected pr-link" and will not substitute the real link. So the changelog validation gate fails, and the GLM-5.2 sweep this entry is meant to trigger cannot be accepted or merged until the link is fixed. AGENTS.md also asks for the real PR URL here.
Verification: The entry at inferencex-e2e/perf-changelog.yaml:11864 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER. PR_LINK_PLACEHOLDERS in validate_perf_changelog.py:21-24 holds only "XXX" and ".../pull/XXX". So the supported reuse merge path aborts, and the link is never rewritten. The PR-run CI check does not fail.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43487923974 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43487923974 |
jiejingzhangamd
added
full-sweep-enabled
jiejingzhangamd
changed the title
Drop the TP8/EP1 concurrency 4 point and its recipe override; concurrency 4 runs on the TP4/EP4 HiCache arm, which delivers more throughput per GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gluon target MoE faults with HSA_STATUS_ERROR_EXCEPTION 0x1016 on full 32768-token uncached prefill chunks (5 crashes on g14/g18). Use chunked-prefill-size 16384 for the TP8/EP1 c1 and c2 points only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d3625a2 to
c0c036a
Compare
Use the 20261111 ROCm 7.2.0 MI35x daily image and Gluon target MoE for GLM-5.2 MXFP4 AgentX MTP at TP8/EP1, concurrency 1 and 2, building on merged #4064. The TP8/EP1 concurrency 4 point is dropped; concurrency 4 runs on the TP4/EP4 HiCache arm, which delivers more throughput per GPU.
The two TP8 variants set
moe-runner-backend: gluonandspeculative-moe-runner-backend: auto. The latter preserves the checkpoint's BF16 NextN draft experts: InferenceX's serve draft as it ships rule prohibits the draft requantization that inheriting Gluon would otherwise perform. The existing EAGLE settings, FP8 KV cache, admission limits, and graph sizes are retained.The shared config image becomes
lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261111(official build; published digestsha256:747da197af01efff6c5dbd1a779dbd24bbb9e22ded91797bbacab0ff175bb55c). TP4/EP4 HiCache and DCP4 retain their recipe settings. Because the image changes for the whole config, the changelog schedules all 12 throughput points. Theall-evalssweep label expands accuracy coverage to both Gluon points as well as the existing TP4 points; default eval selection would cover only TP4/DCP4 C48. The planner deduplicates the two TP4 C16 evals, yielding 11 eval jobs, including both Gluon points.中文
基于已合并的 #4064，使用 20261111 ROCm 7.2.0 MI35x daily image，并将 GLM-5.2 MXFP4 AgentX MTP 的 TP8/EP1、并发 1、2 两个点的 target MoE 切换为 Gluon。去掉 TP8/EP1 并发 4 点，并发 4 由 TP4/EP4 HiCache arm 承担，单卡吞吐更高。
同时显式设置
speculative-moe-runner-backend: auto，保留 checkpoint 中 BF16 NextN draft 专家的原始精度。InferenceX 的 serve draft as it ships 规则禁止 draft 继承 Gluon 后发生的重新量化。EAGLE、FP8 KV cache、请求上限和 graph 大小保持原配置。TP4/EP4 HiCache 与 DCP4 的 recipe 设置保持不变。由于镜像是配置组共用字段，changelog 安排重跑全部 12 个吞吐点，并通过
all-evalssweep 标签扩展准确率评测，确保 Gluon 点也被覆盖；默认评测只会覆盖 TP4/DCP4 C48。规划器会对两个 TP4 C16 评测去重，最终生成 11 个评测任务，包括两个 Gluon 点。AI model disclosure
AI assistance was used to prepare the configuration, recipe changes, and validation. The exact underlying model/version was not exposed and could not be verified.
Updated daily image includes SGLang #48397 (image build-version source
34dfaccc57, digestsha256:747da197af01efff6c5dbd1a779dbd24bbb9e22ded91797bbacab0ff175bb55c). TP8/EP1 concurrency 1 and 2 now usechunked-prefill-size: 32768instead of 16384. This configuration update does not claim new benchmark or accuracy results.Cookbook
The validated MI355X AgentX recipes are now documented in the SGLang GLM-5.2 Cookbook. The documentation update was merged in sgl-project/sglang#50456.
中文
本 PR 验证的 MI355X AgentX 配方现已更新到 SGLang GLM-5.2 Cookbook，对应文档更新已通过 sgl-project/sglang#50456 合并。