Skip to content

Add GLM-5.3-Flash FP8 MI355X SGLang AgentX on the cookbook recipe / 新增基于 cookbook 配方的 GLM-5.3-Flash FP8 MI355X SGLang AgentX - #4720

Merged
Oseltamivir merged 24 commits into
mainfrom
feat/glm53flash-fp8-mi355x-sglang
Nov 6, 2026
Merged

Oseltamivir merged 24 commits into
mainfrom
feat/glm53flash-fp8-mi355x-sglang

Conversation

@functionstackx

@functionstackx functionstackx commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

This PR adds the native FP8 zai-org/GLM-5.3-Flash AgentX lane on MI355X, built on the SGLang cookbook MI355X FP8 Low Latency recipe. It uses the official lmsysorg/sglang:v0.5.22-rocm10-mi35x image, pinned to sha256:1438dd8b9000adaefb5828edb1cc5713992815d4296f4ad1a1e98dee4e9c7e5f, on four GPUs as TP4/EP4 with MTP 5/1/6. The branch was rebuilt on current main. It replaces the earlier stacked lane, which had diverged from the cookbook and never completed a sweep.

Points (8 in total):

  • Concurrency 1, 2, 8, 16, 24 and 32 keep the KV cache in HBM.
  • Concurrency 40 and 48 add the cookbook's HiCache L1 + L2 host tier: --enable-hierarchical-cache --hicache-size 180 --hicache-write-policy write_through, described in the cookbook as the measured MI355X agentic recipe.
  • The agentic eval runs at concurrency 48 with HiCache.
  • Acceptance comes from the committed golden curve glm-5.3-flash__mtp__native__default-default; MTP5 gives an acceptance length of 3.60. This PR sets no acceptance length.

Recipe vs the cookbook

Cookbook flags, used exactly. These come from the MI355X FP8 Low Latency cell plus the deploy panel's default overlays (KV/DSA fp8-tilelang, VLM transport auto, Breakable CUDA Graph off, parsers auto):

  • Environment: SGLANG_USE_AITER=1, AITER_ONLINE_TUNE=0, SGLANG_OPT_FUSED_KDA_VERIFY=1, SGLANG_DSA_FUSE_TOPK=0, SGLANG_ENABLE_WAR_BARRIER=1, SGLANG_FORCE_COARSE_WAR_BARRIER=1.
  • Model and parallelism: --trust-remote-code, --quantization fp8, TP 4, EP 4, --page-size 64.
  • Memory and scheduling: --mem-fraction-static 0.82, --max-running-requests 96, --cuda-graph-max-bs-decode 96, --min-free-slots-delay 1, --chunked-prefill-size 8192, --max-prefill-tokens 8192, --prefill-decode-interval 0.
  • Backends: TileLang DSA prefill and decode, --linear-attn-verify-backend triton, --kv-cache-dtype fp8_e4m3, --moe-runner-backend aiter.
  • Speculative decoding: EAGLE 5/1/6, --speculative-attention-mode prefill, both accept thresholds 1.0, --speculative-moe-a2a-backend none.
  • Other: --mm-feature-transport cpu, --reasoning-parser auto, --tool-call-parser auto, --watchdog-timeout 1800, --dist-timeout 600.
  • HiCache requires sgl-project/sglang#42178. The image's SGLang checkout baa9befa is 586 commits ahead of that PR's merge commit, so the requirement is met.

Deviation required by repository policy: ROCM_QUICK_REDUCE_QUANTIZATION=NONE.

  • The image's config sets ROCM_QUICK_REDUCE_QUANTIZATION=INT8.
  • The MODELS.md all-reduce policy requires SGLang ROCm serving to set NONE explicitly for new models.

Settings the earlier lane set but that already equal v0.5.22 defaults, now removed:

  • --cuda-graph-backend-prefill disabled: default_prefill_backend() already returns disabled on ROCm.
  • --attention-backend dsa: auto-selected for Glm5NextForConditionalGeneration.
  • --linear-attn-backend triton: this is the field default.
  • --context-length 1048576: this equals the checkpoint's max_position_embeddings.
  • SGLANG_GLM_NEXTN_MOE_PTPC=0 and SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0: both default to false.
  • The NextN draft head is served as shipped, with no draft-precision override. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set anywhere.

Checkpoint staging and pinning, removed. These were all harness changes:

  • Removed: the stage-hf-checkpoint.sh setup script, the checkpoint revision pin, the local --tokenizer-path, the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE overrides, the 4-hour health-check window, and the client AIPERF_APPLY_CHAT_TEMPLATE.
  • Instead, zai-org/GLM-5.3-Flash joins the mi355x-amds SHARED_HF_CACHE_LANES policy row, as GLM-5.2-FP8 and DeepSeek-V4.1-Flash already do. The checkpoint resolves from the shared /it-share/hf-hub-cache, which the earlier runs populated.

Harness-only settings kept: served-model-name, enable-metrics, PYTHONNOUSERSITE=1, SGLANG_TIMEOUT_KEEP_ALIVE=900, AIPERF_HTTP_TCP_USER_TIMEOUT, AIPERF_REQUIRED_SERVER_METRIC_PREFIX.

CPU DRAM for HiCache

  • --hicache-size is in units of 10^9 bytes, so the host KV tier is 4 ranks × 180 GB = 720 GB.
  • The cookbook says the total comes to about 190 GB per rank once the host KDA state and DSA indexer pools are included. That is 4 × 190 = 760 GB for the TP4 server.
  • dram-utilization: 0.507 records exactly that: 2,861,022 MiB generator cap × 0.507 × 4/8 GPUs = 725,269 MiB = 760 GB.
  • This is within the MODELS.md proportional limit of 3 TB × 4/8 = 1.5 TB.
  • The host setup hook drops page cache before DRAM points. 760 GB fits on NUMA node 0, where hipHostMalloc pins it.
  • The allocation will be checked against the HiCache server log.

Expected curve (estimate, not a measurement)

Before this PR, only one MI355X point had been measured: concurrency 1 from run 41318690703 (job 124884420596). It measured 4,258 tok/s/GPU, P90 E2E-normalized interactivity 107.7 and P90 E2EL 22.2 s.

That point is only approximate for this recipe. It ran the old recipe with the client chat template on and max-running-requests 2.

The other expected points come from the GB300 FP8 TP4 curves:

  • Sources: 41308011021 without offload and 41352868017 with HiCache. Those runs had no client chat template.
  • Scaling: by the MI355X/GB300 ratio at concurrency 1, which is approximate for the same reason. Throughput ×0.72, E2E-normalized interactivity ×0.85, E2EL ×1.59.
  • Result: 7 expected Pareto points, at concurrency 2, 8, 16, 24 and 32 without offload and 40 and 48 with HiCache. Concurrency 1 is kept as the low-latency anchor.
  • Not run: concurrency 4 and 48 without offload, and concurrency 64 with HiCache, were dominated in the estimate. Concurrency 12 was not dominated but was dropped to save jobs.
  • The sweep below replaces these estimates.

Earlier failure diagnosis

The earlier sweep attempts of run 41318690703 never reached the server.

  • Job 124903440107 (concurrency 16, mia1-p01-g18) and job 124912136191 (concurrency 4, mia1-p01-g15) failed in host_setup.
  • Both hit the error "GPUs still draining prior job's memory after 15min": VRAM stayed at 23–24% against the 10% gate.
  • This is the known mi355x-amds leftover-VRAM drain after a previous job, not a recipe failure.
  • main now waits up to 60 minutes for VRAM to fall to 2% or below (#4813, #4814, #4827), and this branch inherits that hook.
  • Attempt 3 was cancelled manually while queued.
  • The concurrency-1 canary passed with 12 of 258 records dropped (11 warmup and 1 empty stream), consistent with other GLM-5.3-Flash SKUs.
  • The canonical GSM8K eval scored 1281/1319 (97.12%) with ROCM_QUICK_REDUCE_QUANTIZATION=NONE.

Validation

The fresh full sweep for this head is pending. This description will be updated with the run, the measured curve and its Pareto count.

AI model disclosure
  • Claude Opus 5.5 (claude-opus-5-5): rebuilt the lane on the cookbook recipe, produced the estimate and failure diagnosis, and monitors the sweep.
  • Coordinating and independent-review agents: their exact model could not be verified from this agent.
  • GPT-6 Astra medium and GPT-6 Astra: the earlier implementation, research and coordination, still present in the commit history.
中文

本 PR 新增 MI355X 上原生 FP8 zai-org/GLM-5.3-Flash AgentX 配置,采用 SGLang cookbook 的 MI355X FP8 Low Latency 配方。镜像为固定摘要 sha256:1438dd8b9000adaefb5828edb1cc5713992815d4296f4ad1a1e98dee4e9c7e5f 的官方 lmsysorg/sglang:v0.5.22-rocm10-mi35x,在 4 张 GPU 上以 TP4/EP4 运行,使用 MTP 5/1/6。分支已基于当前 main 重建,取代此前偏离 cookbook 且从未完成扫测的叠加版本。

测试点(共 8 个):

  • 并发 1、2、8、16、24、32 的 KV 缓存保留在 HBM 中。
  • 并发 40、48 使用 cookbook 的 HiCache L1 + L2 主机层:--enable-hierarchical-cache --hicache-size 180 --hicache-write-policy write_through,cookbook 将其描述为实测的 MI355X agentic 配方。
  • agentic 评测在并发 48(HiCache)运行。
  • 接受长度取自已提交的黄金曲线 glm-5.3-flash__mtp__native__default-default,MTP5 对应 3.60;本 PR 不设置接受长度。

与 cookbook 的对照:

  • 上文列出的服务参数和环境变量均与 cookbook 的 MI355X FP8 Low Latency 配方及部署面板默认叠加项完全一致。
  • HiCache 依赖的 sgl-project/sglang#42178 已包含在镜像的 SGLang 源码(baa9befa)中。
  • 唯一偏离是仓库政策要求的 ROCM_QUICK_REDUCE_QUANTIZATION=NONE:镜像默认 INT8,而 MODELS.md 的 all-reduce 政策要求新模型在 SGLang ROCm 上显式设为 NONE。
  • 旧版本中与 v0.5.22 默认值相同的设置已删除。NextN 草稿头按发布精度使用,未设置 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。

检查点暂存与固定:

  • 已删除检查点暂存脚本、版本固定、本地 tokenizer 路径、离线变量覆盖、4 小时健康检查窗口及客户端 AIPERF_APPLY_CHAT_TEMPLATE。
  • 改为将 zai-org/GLM-5.3-Flash 加入 mi355x-amds 的 SHARED_HF_CACHE_LANES,与 GLM-5.2-FP8、DeepSeek-V4.1-Flash 一致,检查点从共享的 /it-share/hf-hub-cache 读取。

HiCache 主机内存:

  • --hicache-size 以 10^9 字节为单位:4 × 180 GB = 720 GB 主机 KV。
  • 加上 KDA 状态与 DSA indexer 主机池,cookbook 给出约每 rank 190 GB,即 TP4 服务共 760 GB。
  • dram-utilization: 0.507 正好记录该值:2,861,022 MiB 上限 × 0.507 × 4/8 = 725,269 MiB = 760 GB,低于 MODELS.md 的按比例上限 1.5 TB。

预期曲线(估算,非实测):

  • 此前 MI355X 只有并发 1 一个实测点:4,258 tok/s/GPU,E2E 归一化交互性 P90 为 107.7,E2EL P90 为 22.2 s。
  • 该点用旧配方测得(客户端启用 chat template,max-running-requests 2),对本配方只能作为近似值。
  • 其余点按并发 1 的 MI355X/GB300 近似比值(吞吐 ×0.72、交互性 ×0.85、E2EL ×1.59)由 GB300 FP8 TP4 曲线缩放得到;GB300 锚点运行未启用 chat template。
  • 结果为 7 个预期 Pareto 点:并发 2、8、16、24、32(无卸载)及 40、48(HiCache),并发 1 保留为低延迟锚点。
  • 并发 4、48(无卸载)和 64(HiCache)在估算中被支配,未运行;并发 12 未被支配,为节省任务而未运行。
  • 实测扫测结果将取代这些估算。

此前失败的诊断:

  • 此前扫测在服务启动前于 host_setup 失败:上一任务残留的显存在 15 分钟内一直保持 23–24%,高于 10% 阈值。这是 mi355x-amds 已知的显存回收问题,与配方无关。
  • main 已将等待时间延长到最多 60 分钟、阈值降到 2%,本分支继承了这一修复。
  • 并发 1 canary 已通过,258 条记录中丢弃 12 条(11 条 warmup、1 条空流),与其他 GLM-5.3-Flash SKU 一致。
  • GSM8K 为 1281/1319(97.12%),测试时 ROCM_QUICK_REDUCE_QUANTIZATION=NONE。

验证: 本 head 的全新完整扫测尚在进行,完成后将补充运行链接、实测曲线及 Pareto 点数。

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@functionstackx
functionstackx marked this pull request as ready for review November 1, 2026 11:20
@functionstackx functionstackx added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +1 -0

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@claude

claude Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @functionstackx's task in 50s —— View job


Review

No blocking issues found in the diff.

  • 🟡 docs/MODELS.md needs an entry for glm5.3flash. This PR adds the first active config with that prefix, and docs/MODELS.md doesn't list GLM-5.3-Flash. Repo policy says a new model needs a row there. Add one next to GLM-5.3 (agentic coding, MTP only).

The rest of the diff checks out:

  • The changelog entry is appended at the end of the file and covers glm5.3flash-fp8-mi355x-sglang-agentic-mtp.
  • The image is public on Docker Hub and pinned to a digest.
  • SGLang gets an explicit expert-parallel-size: 4.
  • AIPERF_APPLY_CHAT_TEMPLATE: 'true' is set for MTP.
  • The glm5.3flash_mtp.yaml acceptance curve exists in the base branch.
  • The setup script is in multi_node/srt-slurm-recipes/configs/, the same place other single-node recipes take their setup scripts from.

I couldn't run pytest or infx.matrix.changed: this job isn't allowed to run uv run. The changelog entry covers the only new config key.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread inferencex-e2e/configs/amd-master.yaml Outdated
@@ -1,3 +1,22 @@
glm5.3flash-fp8-mi355x-sglang-agentic-mtp:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new GLM-5.3-Flash lane never gets swept or collected, even with the full-sweep-fail-fast label, because the diff adds no perf-changelog.yaml entry. run-sweep.yml only sweeps PRs that change inferencex-e2e/perf-changelog.yaml, and it builds the matrix from the appended entries. Without an entry, none of the six concurrency rows run. Fix: append one block at the tail of inferencex-e2e/perf-changelog.yaml listing config-keys: [glm5.3flash-fp8-mi355x-sglang-agentic-mtp].

Why this was flagged

The PR adds a new master-config key glm5.3flash-fp8-mi355x-sglang-agentic-mtp (configs/amd-master.yaml:1). It also adds a new recipe, which changes what the benchmark runs. AGENTS.md (Benchmark invariants) says such a change needs a perf-changelog.yaml entry, and that the entry is what triggers result collection. .github/workflows/README.md:165-166 says run-sweep.yml sweeps only same-repository PRs that change inferencex-e2e/perf-changelog.yaml, using the appended entries as the matrix. The diff does not touch perf-changelog.yaml, so the full-sweep-fail-fast label produces no sweep. The six TP4/EP4 points at conc 1/2/4/8/12/16 never run. The PR description says the MXFP4 follow-up waits on this native sweep passing, and that sweep cannot start as the PR stands.

Verification: The trigger is any sweep attempt on this PR, including applying the full-sweep-fail-fast label it already carries. A grep for glm5.3flash in perf-changelog.yaml finds nothing. .github/workflows/run-sweep.yml:19-26 fires on pull_request only for paths: - "inferencex-e2e/perf-changelog.yaml". So with no changelog change, the workflow is never triggered.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This finding targets the initial draft commit 9c2aef0. Commit cd447ad30696c7acd25aa6f93d46037da9c52e3c appended the single actual-PR-URL changelog block before the PR became ready. Current head is 56d8c2ef50991a57cdd2933e6e668c267a24eddc; its Run Sweep has passed changelog/setup/canary selection, and MI355X canary job 124833934093 is now running with nodes:1. No duplicate changelog entry is needed, and this is not yet a GPU or accuracy pass.

中文

此条意见针对初始草稿提交 9c2aef0。提交 cd447ad30696c7acd25aa6f93d46037da9c52e3c 已在 PR 标记为 ready 前追加唯一的、包含实际 PR URL 的 changelog 条目。当前 head 为 56d8c2ef50991a57cdd2933e6e668c267a24eddc;对应的 Run Sweep 已通过 changelog、setup 和 canary 选择,MI355X canary 任务 124833934093 正在运行,并带有 nodes:1 标签。无需重复追加 changelog;目前尚不能认定 GPU 或准确率评测通过。

Comment on lines +21 to +29
available=$(df -PB1 "$HF_HUB_CACHE" | awk 'NR == 2 {print $4}')
cached=0
if [[ -d "$HF_HUB_CACHE/$cache_name" ]]; then
cached=$(du -sB1 "$HF_HUB_CACHE/$cache_name" | awk '{print $1}')
fi
if (( available + cached < HF_CHECKPOINT_BUDGET_BYTES )); then
echo "Insufficient shared-cache disk: available=$available cached=$cached required=$HF_CHECKPOINT_BUDGET_BYTES" >&2
exit 1
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Every GLM-5.3-Flash point fails once the shared cache has under ~72 GB free, even when the checkpoint is already fully cached. The check available + cached < HF_CHECKPOINT_BUDGET_BYTES still demands the full 400 GB after the 328 GB download finishes, so headroom is required on every run. Fix: require free space only for what is still missing, e.g. budget - cached, or skip the check when hf download has nothing to fetch for the pinned revision. That way a complete cache never fails a run.

Why this was flagged

After the first job downloads the checkpoint, cached at stage-hf-checkpoint.sh:24 is about 328 GB. HF_CHECKPOINT_BUDGET_BYTES is '400000000000' (agg-variants.yaml:18), so line 26 fails whenever available drops below about 72 GB. /it-share/hf-hub-cache is shared with the other SHARED_HF_CACHE_LANES models (MiniMax-M3.1, GLM-5.2-FP8, DeepSeek-V4.1-Flash, Muse-Spark), so free space shrinks as they get cached. When that happens, all six concurrency points exit 1 in setup with "Insufficient shared-cache disk". Nothing needed downloading: hf download at the pinned revision would have been a no-op. No safeguard covers this: the check never asks whether the snapshot for HF_CHECKPOINT_REVISION is already complete.

Verification: stage-hf-checkpoint.sh:21-26 fails if (( available + cached < HF_CHECKPOINT_BUDGET_BYTES )), with a budget of '400000000000' (agg-variants.yaml:18). Once the checkpoint is fully cached, cached is about 328 GB, so about 72 GB must still be free, even though hf download (lines 30-32) would fetch nothing. Every run would then exit 1 at line 27 with "Insufficient shared-cache disk".

Comment on lines +30 to +32
hf download "$HF_CHECKPOINT_REPO" \
--revision "$HF_CHECKPOINT_REVISION" \
--cache-dir "$HF_HUB_CACHE"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) One transient error during the 328 GB download fails the whole point, and the sibling points queued on the lock retry the download in turn. The script runs hf download once with no retry. The Kimi staging script for this same cluster's /it-share cache, kimik3-atom-dspark-draft.sh, retries five times because "Shared-cache downloads can hit transient stale handles". Fix: retry hf download a bounded number of times with a backoff while still holding the flock, and fail clearly after the last attempt.

Why this was flagged

The new lane runs on cluster:mi355x-amds with the shared cache at /it-share/hf-hub-cache (configs/runners.yaml shared-hf-hub-cache). It downloads a 328 GB checkpoint with a single hf download at stage-hf-checkpoint.sh:30-32 under set -eo pipefail. Any transient error makes the setup script exit non-zero, so that concurrency point fails. kimik3-atom-dspark-draft.sh:2-12 shows this error class is known on this cache: it retries a 7 GB download five times with a 60 s sleep for that reason. The new script has no retry. Each of the six points waiting on the flock then starts its own full attempt, and each can fail the same way.

Verification: stage-hf-checkpoint.sh:30-32 runs a single hf download "$HF_CHECKPOINT_REPO" --revision ... --cache-dir "$HF_HUB_CACHE" with no retry loop, and line 2 sets set -eo pipefail, so any non-zero exit fails the setup script and with it that concurrency point. The sibling kimik3-atom-dspark-draft.sh retries 5 times with 60 s sleeps.

@Oseltamivir
Oseltamivir changed the base branch from feat/glm53flash-golden-al to main November 5, 2026 11:25
…kbook recipe

Rebuild the MI355X FP8 lane on current main with the cookbook MI355X FP8
Low Latency recipe: TP4/EP4, HBM-only KV at conc 1-32 and the cookbook
HiCache L1 + L2 tier at conc 40, 48 and 64.
@Oseltamivir Oseltamivir changed the title [DNM, Experimental] Add GLM-5.3-Flash native FP8 MI355X benchmarks / [DNM, Experimental] 新增 GLM-5.3-Flash 原生 FP8 MI355X 基准测试 Add GLM-5.3-Flash FP8 MI355X SGLang AgentX on the cookbook recipe / 新增基于 cookbook 配方的 GLM-5.3-Flash FP8 MI355X SGLang AgentX Nov 5, 2026
@Oseltamivir Oseltamivir added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 5, 2026
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@Oseltamivir

Copy link
Copy Markdown
Collaborator

/use 42074044563

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

@Oseltamivir staged run 42074044563.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…e size

The c40/c48 HiCache servers allocate 187.33 GB of host memory per rank
(749.33 GB per TP4 server); record 751 GB. SGLang v0.5.22 forces page
size 256 for DSA models, so state it.
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml

@Oseltamivir Oseltamivir left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core approval after checking the work.

  • Sweep run 42074044563 (selected via /use) finished green on swept head 51a9cf1 (the c8 point passed on attempt 4). FP8 TP4/EP4 MTP 5/1/6, GPU-resident KV at c1–c32 and the cookbook HiCache tier (hicache-size 180, write-through) at c40/c48.
  • I checked SGLang v0.5.22 and the resolved server args from the c48 log for lossy defaults; none apply:
    • DSA indexer skipping is driven only by the checkpoint's indexer_types, and all 45 GLM-5.3-Flash layers are full.
    • Dense prefill applies at kv_len ≤ index_topk=2048, which is exact.
    • The gfx950 fused DSA indexer is shape-gated to hidden 6144 / q_lora 2048; this model is 4096 / 1536, so the standard path runs.
    • HiSparse is off; speculative accept thresholds are 1.0/1.0 with rejection sampling and adaptive speculation off; there is no stochastic or int8 linear-attention state.
    • ROCM_QUICK_REDUCE_QUANTIZATION=NONE overrides the image's INT8 all-reduce.
    • FP8 KV is the cookbook's explicit setting.
  • Since the sweep, the diff changes only metadata:
    • dram-utilization 0.507 → 0.501, recorded as 751 GB. The c40 and c48 logs both show 187.33 GB per rank (113.10 KV + 66.94 linear-attention state + 7.29 DSA indexer, decimal GB), 749.33 GB for TP4.
    • page-size 64 → 256, a no-op: the run already resolved page_size=256.
    • Comment and changelog wording.
  • Head 6f21ca5 merges the latest main, with the MI355X changelog block last. Reuse gate run 42210414376 skipped every sweep ("PR #4720 has /reuse-sweep-run authorization"). Lint and Tests pass.

Core 审批,已核对。

  • /use 指定的扫测运行 42074044563 在 51a9cf1 上全部通过(c8 在第 4 次尝试时通过)。FP8 TP4/EP4 MTP 5/1/6,c1–c32 的 KV 全部驻留 GPU,c40/c48 使用 cookbook 的 HiCache 层(hicache-size 180,write-through)。
  • 我查看了 SGLang v0.5.22 源码和 c48 日志中实际生效的服务参数,确认没有开启任何有损默认项:
    • DSA indexer 是否跳过只由权重配置中的 indexer_types 决定,GLM-5.3-Flash 的 45 层全部为 full。
    • kv_len ≤ index_topk=2048 时走稠密预填充,结果完全等价。
    • gfx950 融合 indexer 只适用于 hidden 6144 / q_lora 2048 的形状;本模型为 4096 / 1536,因此走标准路径。
    • HiSparse 关闭;投机接受阈值为 1.0/1.0,拒绝采样和自适应投机均关闭;线性注意力状态没有随机舍入或 int8。
    • ROCM_QUICK_REDUCE_QUANTIZATION=NONE 覆盖了镜像默认的 INT8 all-reduce。
    • FP8 KV 是 cookbook 明确指定的设置。
  • 扫测之后只改了元数据:
    • dram-utilization 从 0.507 改为 0.501,记为 751 GB。c40 与 c48 日志均为每 rank 187.33 GB(113.10 KV + 66.94 线性注意力状态 + 7.29 DSA indexer,十进制 GB),TP4 合计 749.33 GB。
    • page-size 从 64 改为 256,没有实际影响:运行时已解析为 page_size=256。
    • 注释与 changelog 措辞。
  • 6f21ca5 已合入最新 main,MI355X 的 changelog 块位于末尾。复用门禁运行 42210414376 跳过了所有扫测("PR #4720 has /reuse-sweep-run authorization")。Lint 与 Tests 均通过。

@Oseltamivir

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 42074044563

@Oseltamivir
Oseltamivir merged commit 1e7c8c9 into main Nov 6, 2026
33 checks passed
@Oseltamivir
Oseltamivir deleted the feat/glm53flash-fp8-mi355x-sglang branch November 6, 2026 11:47
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 8 0 0 0 0
eval 1 1 1 0 0
trace 8 0 0 0 0
power 8 0 0 0 0

benchmarkFiles 8, rawBenchmarkRows 8, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 1319, gpuMetricSamplesNew 151496, powerPublicationPoints 8, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants