Repository navigation
Add GLM-5.3-Flash FP8 MI355X SGLang AgentX on the cookbook recipe / 新增基于 cookbook 配方的 GLM-5.3-Flash FP8 MI355X SGLang AgentX - #4720
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
|
Claude finished @functionstackx's task in 50s —— View job ReviewNo blocking issues found in the diff.
The rest of the diff checks out:
I couldn't run |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42074044563 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42074044563 |
…erplexity.ai/SemiAnalysisAI/InferenceX into feat/glm53flash-fp8-mi355x-sglang
| @@ -1,3 +1,22 @@ | |||
| glm5.3flash-fp8-mi355x-sglang-agentic-mtp: | |||
There was a problem hiding this comment.
🔴 The new GLM-5.3-Flash lane never gets swept or collected, even with the full-sweep-fail-fast label, because the diff adds no perf-changelog.yaml entry. run-sweep.yml only sweeps PRs that change inferencex-e2e/perf-changelog.yaml, and it builds the matrix from the appended entries. Without an entry, none of the six concurrency rows run. Fix: append one block at the tail of inferencex-e2e/perf-changelog.yaml listing config-keys: [glm5.3flash-fp8-mi355x-sglang-agentic-mtp].
Why this was flagged
The PR adds a new master-config key glm5.3flash-fp8-mi355x-sglang-agentic-mtp (configs/amd-master.yaml:1). It also adds a new recipe, which changes what the benchmark runs. AGENTS.md (Benchmark invariants) says such a change needs a perf-changelog.yaml entry, and that the entry is what triggers result collection. .github/workflows/README.md:165-166 says run-sweep.yml sweeps only same-repository PRs that change inferencex-e2e/perf-changelog.yaml, using the appended entries as the matrix. The diff does not touch perf-changelog.yaml, so the full-sweep-fail-fast label produces no sweep. The six TP4/EP4 points at conc 1/2/4/8/12/16 never run. The PR description says the MXFP4 follow-up waits on this native sweep passing, and that sweep cannot start as the PR stands.
Verification: The trigger is any sweep attempt on this PR, including applying the full-sweep-fail-fast label it already carries. A grep for glm5.3flash in perf-changelog.yaml finds nothing. .github/workflows/run-sweep.yml:19-26 fires on pull_request only for paths: - "inferencex-e2e/perf-changelog.yaml". So with no changelog change, the workflow is never triggered.
There was a problem hiding this comment.
This finding targets the initial draft commit 9c2aef0. Commit cd447ad30696c7acd25aa6f93d46037da9c52e3c appended the single actual-PR-URL changelog block before the PR became ready. Current head is 56d8c2ef50991a57cdd2933e6e668c267a24eddc; its Run Sweep has passed changelog/setup/canary selection, and MI355X canary job 124833934093 is now running with nodes:1. No duplicate changelog entry is needed, and this is not yet a GPU or accuracy pass.
中文
此条意见针对初始草稿提交 9c2aef0。提交 cd447ad30696c7acd25aa6f93d46037da9c52e3c 已在 PR 标记为 ready 前追加唯一的、包含实际 PR URL 的 changelog 条目。当前 head 为 56d8c2ef50991a57cdd2933e6e668c267a24eddc;对应的 Run Sweep 已通过 changelog、setup 和 canary 选择,MI355X canary 任务 124833934093 正在运行,并带有 nodes:1 标签。无需重复追加 changelog;目前尚不能认定 GPU 或准确率评测通过。
| available=$(df -PB1 "$HF_HUB_CACHE" | awk 'NR == 2 {print $4}') | ||
| cached=0 | ||
| if [[ -d "$HF_HUB_CACHE/$cache_name" ]]; then | ||
| cached=$(du -sB1 "$HF_HUB_CACHE/$cache_name" | awk '{print $1}') | ||
| fi | ||
| if (( available + cached < HF_CHECKPOINT_BUDGET_BYTES )); then | ||
| echo "Insufficient shared-cache disk: available=$available cached=$cached required=$HF_CHECKPOINT_BUDGET_BYTES" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
🟡 (optional) Every GLM-5.3-Flash point fails once the shared cache has under ~72 GB free, even when the checkpoint is already fully cached. The check available + cached < HF_CHECKPOINT_BUDGET_BYTES still demands the full 400 GB after the 328 GB download finishes, so headroom is required on every run. Fix: require free space only for what is still missing, e.g. budget - cached, or skip the check when hf download has nothing to fetch for the pinned revision. That way a complete cache never fails a run.
Why this was flagged
After the first job downloads the checkpoint, cached at stage-hf-checkpoint.sh:24 is about 328 GB. HF_CHECKPOINT_BUDGET_BYTES is '400000000000' (agg-variants.yaml:18), so line 26 fails whenever available drops below about 72 GB. /it-share/hf-hub-cache is shared with the other SHARED_HF_CACHE_LANES models (MiniMax-M3.1, GLM-5.2-FP8, DeepSeek-V4.1-Flash, Muse-Spark), so free space shrinks as they get cached. When that happens, all six concurrency points exit 1 in setup with "Insufficient shared-cache disk". Nothing needed downloading: hf download at the pinned revision would have been a no-op. No safeguard covers this: the check never asks whether the snapshot for HF_CHECKPOINT_REVISION is already complete.
Verification: stage-hf-checkpoint.sh:21-26 fails if (( available + cached < HF_CHECKPOINT_BUDGET_BYTES )), with a budget of '400000000000' (agg-variants.yaml:18). Once the checkpoint is fully cached, cached is about 328 GB, so about 72 GB must still be free, even though hf download (lines 30-32) would fetch nothing. Every run would then exit 1 at line 27 with "Insufficient shared-cache disk".
| hf download "$HF_CHECKPOINT_REPO" \ | ||
| --revision "$HF_CHECKPOINT_REVISION" \ | ||
| --cache-dir "$HF_HUB_CACHE" |
There was a problem hiding this comment.
🟡 (optional) One transient error during the 328 GB download fails the whole point, and the sibling points queued on the lock retry the download in turn. The script runs hf download once with no retry. The Kimi staging script for this same cluster's /it-share cache, kimik3-atom-dspark-draft.sh, retries five times because "Shared-cache downloads can hit transient stale handles". Fix: retry hf download a bounded number of times with a backoff while still holding the flock, and fail clearly after the last attempt.
Why this was flagged
The new lane runs on cluster:mi355x-amds with the shared cache at /it-share/hf-hub-cache (configs/runners.yaml shared-hf-hub-cache). It downloads a 328 GB checkpoint with a single hf download at stage-hf-checkpoint.sh:30-32 under set -eo pipefail. Any transient error makes the setup script exit non-zero, so that concurrency point fails. kimik3-atom-dspark-draft.sh:2-12 shows this error class is known on this cache: it retries a 7 GB download five times with a 60 s sleep for that reason. The new script has no retry. Each of the six points waiting on the flock then starts its own full attempt, and each can fail the same way.
Verification: stage-hf-checkpoint.sh:30-32 runs a single hf download "$HF_CHECKPOINT_REPO" --revision ... --cache-dir "$HF_HUB_CACHE" with no retry loop, and line 2 sets set -eo pipefail, so any non-zero exit fails the setup script and with it that concurrency point. The sibling kimik3-atom-dspark-draft.sh retries 5 times with 60 s sleeps.
…kbook recipe Rebuild the MI355X FP8 lane on current main with the cookbook MI355X FP8 Low Latency recipe: TP4/EP4, HBM-only KV at conc 1-32 and the cookbook HiCache L1 + L2 tier at conc 40, 48 and 64.
# Conflicts: # inferencex-e2e/perf-changelog.yaml
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/use 42074044563 |
|
@Oseltamivir staged run 42074044563. This run remains available across future |
…e size The c40/c48 HiCache servers allocate 187.33 GB of host memory per rank (749.33 GB per TP4 server); record 751 GB. SGLang v0.5.22 forces page size 256 for DSA models, so state it.
# Conflicts: # inferencex-e2e/perf-changelog.yaml
Oseltamivir
left a comment
There was a problem hiding this comment.
Core approval after checking the work.
- Sweep run 42074044563 (selected via
/use) finished green on swept head 51a9cf1 (the c8 point passed on attempt 4). FP8 TP4/EP4 MTP 5/1/6, GPU-resident KV at c1–c32 and the cookbook HiCache tier (hicache-size 180, write-through) at c40/c48. - I checked SGLang v0.5.22 and the resolved server args from the c48 log for lossy defaults; none apply:
- DSA indexer skipping is driven only by the checkpoint's
indexer_types, and all 45 GLM-5.3-Flash layers arefull. - Dense prefill applies at kv_len ≤ index_topk=2048, which is exact.
- The gfx950 fused DSA indexer is shape-gated to hidden 6144 / q_lora 2048; this model is 4096 / 1536, so the standard path runs.
- HiSparse is off; speculative accept thresholds are 1.0/1.0 with rejection sampling and adaptive speculation off; there is no stochastic or int8 linear-attention state.
ROCM_QUICK_REDUCE_QUANTIZATION=NONEoverrides the image's INT8 all-reduce.- FP8 KV is the cookbook's explicit setting.
- DSA indexer skipping is driven only by the checkpoint's
- Since the sweep, the diff changes only metadata:
dram-utilization0.507 → 0.501, recorded as 751 GB. The c40 and c48 logs both show 187.33 GB per rank (113.10 KV + 66.94 linear-attention state + 7.29 DSA indexer, decimal GB), 749.33 GB for TP4.page-size64 → 256, a no-op: the run already resolvedpage_size=256.- Comment and changelog wording.
- Head 6f21ca5 merges the latest main, with the MI355X changelog block last. Reuse gate run 42210414376 skipped every sweep ("PR #4720 has /reuse-sweep-run authorization"). Lint and Tests pass.
Core 审批,已核对。
/use指定的扫测运行 42074044563 在 51a9cf1 上全部通过(c8 在第 4 次尝试时通过)。FP8 TP4/EP4 MTP 5/1/6,c1–c32 的 KV 全部驻留 GPU,c40/c48 使用 cookbook 的 HiCache 层(hicache-size 180,write-through)。- 我查看了 SGLang v0.5.22 源码和 c48 日志中实际生效的服务参数,确认没有开启任何有损默认项:
- DSA indexer 是否跳过只由权重配置中的
indexer_types决定,GLM-5.3-Flash 的 45 层全部为full。 - kv_len ≤ index_topk=2048 时走稠密预填充,结果完全等价。
- gfx950 融合 indexer 只适用于 hidden 6144 / q_lora 2048 的形状;本模型为 4096 / 1536,因此走标准路径。
- HiSparse 关闭;投机接受阈值为 1.0/1.0,拒绝采样和自适应投机均关闭;线性注意力状态没有随机舍入或 int8。
ROCM_QUICK_REDUCE_QUANTIZATION=NONE覆盖了镜像默认的 INT8 all-reduce。- FP8 KV 是 cookbook 明确指定的设置。
- DSA indexer 是否跳过只由权重配置中的
- 扫测之后只改了元数据:
dram-utilization从 0.507 改为 0.501,记为 751 GB。c40 与 c48 日志均为每 rank 187.33 GB(113.10 KV + 66.94 线性注意力状态 + 7.29 DSA indexer,十进制 GB),TP4 合计 749.33 GB。page-size从 64 改为 256,没有实际影响:运行时已解析为page_size=256。- 注释与 changelog 措辞。
- 6f21ca5 已合入最新 main,MI355X 的 changelog 块位于末尾。复用门禁运行 42210414376 跳过了所有扫测("PR #4720 has /reuse-sweep-run authorization")。Lint 与 Tests 均通过。
|
/reuse-sweep-run 42074044563 |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 8, rawBenchmarkRows 8, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 1319, gpuMetricSamplesNew 151496, powerPublicationPoints 8, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
This PR adds the native FP8
zai-org/GLM-5.3-FlashAgentX lane on MI355X, built on the SGLang cookbook MI355X FP8 Low Latency recipe. It uses the officiallmsysorg/sglang:v0.5.22-rocm10-mi35ximage, pinned tosha256:1438dd8b9000adaefb5828edb1cc5713992815d4296f4ad1a1e98dee4e9c7e5f, on four GPUs as TP4/EP4 with MTP 5/1/6. The branch was rebuilt on currentmain. It replaces the earlier stacked lane, which had diverged from the cookbook and never completed a sweep.Points (8 in total):
--enable-hierarchical-cache --hicache-size 180 --hicache-write-policy write_through, described in the cookbook as the measured MI355X agentic recipe.glm-5.3-flash__mtp__native__default-default; MTP5 gives an acceptance length of 3.60. This PR sets no acceptance length.Recipe vs the cookbook
Cookbook flags, used exactly. These come from the MI355X FP8 Low Latency cell plus the deploy panel's default overlays (KV/DSA
fp8-tilelang, VLM transport auto, Breakable CUDA Graph off, parsers auto):SGLANG_USE_AITER=1,AITER_ONLINE_TUNE=0,SGLANG_OPT_FUSED_KDA_VERIFY=1,SGLANG_DSA_FUSE_TOPK=0,SGLANG_ENABLE_WAR_BARRIER=1,SGLANG_FORCE_COARSE_WAR_BARRIER=1.--trust-remote-code,--quantization fp8, TP 4, EP 4,--page-size 64.--mem-fraction-static 0.82,--max-running-requests 96,--cuda-graph-max-bs-decode 96,--min-free-slots-delay 1,--chunked-prefill-size 8192,--max-prefill-tokens 8192,--prefill-decode-interval 0.--linear-attn-verify-backend triton,--kv-cache-dtype fp8_e4m3,--moe-runner-backend aiter.--speculative-attention-mode prefill, both accept thresholds 1.0,--speculative-moe-a2a-backend none.--mm-feature-transport cpu,--reasoning-parser auto,--tool-call-parser auto,--watchdog-timeout 1800,--dist-timeout 600.baa9befais 586 commits ahead of that PR's merge commit, so the requirement is met.Deviation required by repository policy:
ROCM_QUICK_REDUCE_QUANTIZATION=NONE.ROCM_QUICK_REDUCE_QUANTIZATION=INT8.NONEexplicitly for new models.Settings the earlier lane set but that already equal v0.5.22 defaults, now removed:
--cuda-graph-backend-prefill disabled:default_prefill_backend()already returnsdisabledon ROCm.--attention-backend dsa: auto-selected forGlm5NextForConditionalGeneration.--linear-attn-backend triton: this is the field default.--context-length 1048576: this equals the checkpoint'smax_position_embeddings.SGLANG_GLM_NEXTN_MOE_PTPC=0andSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0: both default to false.SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not set anywhere.Checkpoint staging and pinning, removed. These were all harness changes:
stage-hf-checkpoint.shsetup script, the checkpoint revision pin, the local--tokenizer-path, theHF_HUB_OFFLINE/TRANSFORMERS_OFFLINEoverrides, the 4-hour health-check window, and the clientAIPERF_APPLY_CHAT_TEMPLATE.zai-org/GLM-5.3-Flashjoins the mi355x-amdsSHARED_HF_CACHE_LANESpolicy row, as GLM-5.2-FP8 and DeepSeek-V4.1-Flash already do. The checkpoint resolves from the shared/it-share/hf-hub-cache, which the earlier runs populated.Harness-only settings kept:
served-model-name,enable-metrics,PYTHONNOUSERSITE=1,SGLANG_TIMEOUT_KEEP_ALIVE=900,AIPERF_HTTP_TCP_USER_TIMEOUT,AIPERF_REQUIRED_SERVER_METRIC_PREFIX.CPU DRAM for HiCache
--hicache-sizeis in units of 10^9 bytes, so the host KV tier is 4 ranks × 180 GB = 720 GB.dram-utilization: 0.507records exactly that: 2,861,022 MiB generator cap × 0.507 × 4/8 GPUs = 725,269 MiB = 760 GB.hipHostMallocpins it.Expected curve (estimate, not a measurement)
Before this PR, only one MI355X point had been measured: concurrency 1 from run 41318690703 (job 124884420596). It measured 4,258 tok/s/GPU, P90 E2E-normalized interactivity 107.7 and P90 E2EL 22.2 s.
That point is only approximate for this recipe. It ran the old recipe with the client chat template on and
max-running-requests 2.The other expected points come from the GB300 FP8 TP4 curves:
Earlier failure diagnosis
The earlier sweep attempts of run 41318690703 never reached the server.
mia1-p01-g18) and job 124912136191 (concurrency 4,mia1-p01-g15) failed inhost_setup.mainnow waits up to 60 minutes for VRAM to fall to 2% or below (#4813, #4814, #4827), and this branch inherits that hook.ROCM_QUICK_REDUCE_QUANTIZATION=NONE.Validation
The fresh full sweep for this head is pending. This description will be updated with the run, the measured curve and its Pareto count.
AI model disclosure
claude-opus-5-5): rebuilt the lane on the cookbook recipe, produced the estimate and failure diagnosis, and monitors the sweep.中文
本 PR 新增 MI355X 上原生 FP8
zai-org/GLM-5.3-FlashAgentX 配置,采用 SGLang cookbook 的 MI355X FP8 Low Latency 配方。镜像为固定摘要sha256:1438dd8b9000adaefb5828edb1cc5713992815d4296f4ad1a1e98dee4e9c7e5f的官方lmsysorg/sglang:v0.5.22-rocm10-mi35x,在 4 张 GPU 上以 TP4/EP4 运行,使用 MTP 5/1/6。分支已基于当前main重建,取代此前偏离 cookbook 且从未完成扫测的叠加版本。测试点(共 8 个):
--enable-hierarchical-cache --hicache-size 180 --hicache-write-policy write_through,cookbook 将其描述为实测的 MI355X agentic 配方。glm-5.3-flash__mtp__native__default-default,MTP5 对应 3.60;本 PR 不设置接受长度。与 cookbook 的对照:
baa9befa)中。ROCM_QUICK_REDUCE_QUANTIZATION=NONE:镜像默认INT8,而 MODELS.md 的 all-reduce 政策要求新模型在 SGLang ROCm 上显式设为NONE。SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。检查点暂存与固定:
AIPERF_APPLY_CHAT_TEMPLATE。zai-org/GLM-5.3-Flash加入 mi355x-amds 的SHARED_HF_CACHE_LANES,与 GLM-5.2-FP8、DeepSeek-V4.1-Flash 一致,检查点从共享的/it-share/hf-hub-cache读取。HiCache 主机内存:
--hicache-size以 10^9 字节为单位:4 × 180 GB = 720 GB 主机 KV。dram-utilization: 0.507正好记录该值:2,861,022 MiB 上限 × 0.507 × 4/8 = 725,269 MiB = 760 GB,低于 MODELS.md 的按比例上限 1.5 TB。预期曲线(估算,非实测):
max-running-requests 2),对本配方只能作为近似值。此前失败的诊断:
host_setup失败:上一任务残留的显存在 15 分钟内一直保持 23–24%,高于 10% 阈值。这是 mi355x-amds 已知的显存回收问题,与配方无关。main已将等待时间延长到最多 60 分钟、阈值降到 2%,本分支继承了这一修复。ROCM_QUICK_REDUCE_QUANTIZATION=NONE。验证: 本 head 的全新完整扫测尚在进行,完成后将补充运行链接、实测曲线及 Pareto 点数。