Repository navigation
Add GLM-5.3-Flash FP8 and NVFP4 B300 SGLang AgentX / 添加 GLM-5.3-Flash FP8 与 NVFP4 B300 SGLang AgentX - #4710
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
functionstackx
added
the
full-sweep-fail-fast
…erplexity.ai/SemiAnalysisAI/InferenceX into feat/glm53flash-fp8-b300-sglang
…erplexity.ai/SemiAnalysisAI/InferenceX into feat/glm53flash-fp8-b300-sglang
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42024862281 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42024862281 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| glm5.3flash-fp8-b300-sglang-agentic-mtp: | ||
| image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae | ||
| model: zai-org/GLM-5.3-Flash | ||
| model-prefix: glm5.3flash | ||
| runner: cluster:b300-dsxe | ||
| precision: fp8 | ||
| framework: sglang | ||
| multinode: false | ||
| srt-recipe-dir: glm5.3flash/sglang/b300-fp8-mtp | ||
| scenarios: | ||
| agentic-coding: | ||
| - search-space: | ||
| - { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 24, 32, 48, 64], srt-recipe: agentic.yaml } |
There was a problem hiding this comment.
🔴 Results for the new GLM-5.3-Flash B300 AgentX config will never be collected, because no perf-changelog.yaml entry names it. The PR adds config key glm5.3flash-fp8-b300-sglang-agentic-mtp and its recipe, but inferencex-e2e/perf-changelog.yaml is not changed. Under AGENTS.md, that changelog entry is what starts result collection and gates /use before merge. Fix: append one block at the end of inferencex-e2e/perf-changelog.yaml with config-keys: [glm5.3flash-fp8-b300-sglang-agentic-mtp], scenario-type: [agentic-coding], an English description and pr-link, as was done for glm5.3-fp4-b300-sglang-agentic-mtp.
Why this was flagged
The diff adds the master-config key glm5.3flash-fp8-b300-sglang-agentic-mtp (nvidia-master.yaml:8823) and the recipe glm5.3flash/sglang/b300-fp8-mtp/agentic.yaml. That changes what benchmarks run, and python -m infx.matrix.changed would list the new key. AGENTS.md (Benchmark invariants) says such a change needs a perf-changelog.yaml entry, and that the entry is what triggers result collection. The diff adds no entry to inferencex-e2e/perf-changelog.yaml. The sibling glm5.3-fp4-b300-sglang-agentic-mtp has one at perf-changelog.yaml:10869-10875. Without it, the sweep is never triggered or ingested for this config, and the merge gate (/use <run_id> via merge_with_reuse) has nothing to check, so the PR can merge with a config that never produces results.
Verification: Trigger: the PR merges as it stands. AGENTS.md says at line 87: "A change that alters what any benchmark runs needs a perf-changelog.yaml entry for the affected config keys; the entry triggers result collection." Without a block, the sweep-and-ingest flow tied to the changelog never collects results for glm5.3flash-fp8-b300-sglang-agentic-mtp.
There was a problem hiding this comment.
Addressed in 83a606a before applying the sweep label. The current head 3bb2eb3 includes one append-only perf-changelog block naming glm5.3flash-fp8-b300-sglang-agentic-mtp with this PR URL. Local changelog validation passes, and the exact-head sweep is running: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41300756502. This finding references the earlier 6cca971 commit.
| glm5.3flash-fp8-b300-sglang-agentic-mtp: | ||
| image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae | ||
| model: zai-org/GLM-5.3-Flash | ||
| model-prefix: glm5.3flash |
There was a problem hiding this comment.
🟡 (optional) GLM-5.3-Flash sweep jobs get the lowest model priority in the CI queue, behind every other active model. The new model-prefix: glm5.3flash is not registered where the scheduler and docs look up prefixes. calculate_priority matches model-prefix exactly, so glm5.3flash gets +0 where glm5.3 gets +0.5 and dsr1 gets +0.05. Fix: register the new prefix everywhere prefixes are listed, which covers the 3 sites listed. Add glm5.3flash to ci-priority.yaml model-prefix and to the run-sweep.yml classifier enum, which test_ci_priority.py:252 requires. Add a GLM-5.3-Flash row to MODELS.md. Same pattern at 3 sites (configs/ci-priority.yaml:30, .github/workflows/run-sweep.yml:246, docs/MODELS.md:125).
Why this was flagged
Every matrix job generated from glm5.3flash-fp8-b300-sglang-agentic-mtp carries model-prefix: glm5.3flash (nvidia-master.yaml:8826). On push events, calculate_priority (infx/workflows/ci_priority.py:180-182) looks up adjustments.model-prefix["glm5.3flash"]. That key is absent from configs/ci-priority.yaml:30-41, so the adjustment is 0. On PR sweeps, the classifier enum at .github/workflows/run-sweep.yml:246 has no glm5.3flash. Even if the classifier returns glm5.3, _entry_from_criteria (ci_priority.py:132-140) only matches == criterion or startswith(criterion + "-"), so it projects the prefix to "". docs/MODELS.md:154 also says "Adding a model? ... add a row here in the same PR", but the Model support matrix has no glm5.3flash row. No validator rejects an unregistered prefix, so nothing catches the omission.
Verification: Trigger: every matrix job generated from the new entry, on push and on PR runs, gets no model-prefix bonus. The new entry sets model-prefix: glm5.3flash (nvidia-master.yaml:8826). configs/ci-priority.yaml:30-41 has no glm5.3flash, so the adjustment is 0, while the sibling glm5.3 gets 0.5.
functionstackx
changed the title
|
/use 41301274173 |
|
@Oseltamivir staging run 41301274173. Existing staged runs will be preserved; a completion link will be posted here. |
Serve both precisions with the SGLang cookbook's B300 Low Latency flags: GPU-resident KV at concurrency up to 48, and at concurrency 64 and 96 HiCache with auto-sized host pools (fraction 0.76, within the 3 TB node cap) and max-running-requests 96, which SGLang otherwise resets to 48 under EAGLE.
Oseltamivir
changed the title
Oseltamivir
added
full-sweep-fail-fast
|
/use 42024862281 |
|
@Oseltamivir staging run 42024862281. Existing staged runs will be preserved; a completion link will be posted here. |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
Run 42024862281 allocated up to 2802.2 GB (FP8) and 2813.6 GB (NVFP4) of HiCache host pools per node; dram-utilization 0.935 and 0.939 record 2804 and 2816 GB.
Oseltamivir
left a comment
Oseltamivir
left a comment
There was a problem hiding this comment.
Core approval after checking the work.
- Sweep run 42024862281 (selected via
/use) finished green on swept head 88199b2: FP8 and NVFP4, base points at low concurrency and HiCache at high concurrency, with no TP4 on the 8-GPU node. - PR-own diff 88199b2 → 216f4da changes only
dram-utilization: FP8 0.933667 → 0.935 and NVFP4 0.937334 → 0.939. The generator records 2804/2816 GB, ≥ the measured 2802.2/2813.6 GB. No server or recipe arguments changed. - Push run 42147521719 passed the reuse gate and skipped every benchmark.
- The reviewer agent independently confirmed the same.
- The only conflict with main is the perf-changelog tail, which merge_with_reuse resolves.
Core 审批，已核对。
/use指定的扫测运行 42024862281 在 88199b2 上全部通过：FP8 与 NVFP4 两条线，低并发为基础点，高并发为 HiCache；8 卡节点不跑 TP4。- 88199b2 → 216f4da 之间本 PR 只改了
dram-utilization：FP8 从 0.933667 改为 0.935，NVFP4 从 0.937334 改为 0.939。生成器记为 2804/2816 GB，不低于实测的 2802.2/2813.6 GB。服务器和 recipe 参数均未改动。 - 推送后的运行 42147521719 通过复用门禁，所有基准任务均被跳过。
- 审查代理独立核对，结论一致。
- 与 main 的唯一冲突在 perf-changelog 末尾，由 merge_with_reuse 自动解决。
|
/reuse-sweep-run 42024862281 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 18, rawBenchmarkRows 34, coveredByPerJobFiles 17, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 901864, powerPublicationPoints 17, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds GLM-5.3-Flash AgentX on one 8x B300 node for both precisions, in one PR:
glm5.3flash-fp8-b300-sglang-agentic-mtpzai-org/GLM-5.3-Flashglm5.3flash-fp4-b300-sglang-agentic-mtpnvidia/GLM-5.3-Flash-NVFP4It replaces the separate B300 PRs:
Both precisions serve the SGLang cookbook's B300 Low Latency cells (source) on the official
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae:--quantization modelopt_fp4.Everything else the earlier recipes tuned is now the engine default: memory fraction, chunked/max prefill, decode CUDA graph sizes and admission on the no-offload points. Acceptance comes from the
glm-5.3-flash__mtp__native__default-defaultgolden curve (MTP5 = 3.60).Results: sweep 42024862281
The run measured head
88199b245. It completed green on attempt 1, with every pointpower_valid=1. The commit after it changes only the recorded DRAM metadata (see below).Total tok/s per GPU / p90 E2EL. ★ = on the throughput vs p90 E2EL Pareto front.
infx.workflows.pareto_coverageon these points: FP8 has 7 frontier points of 9, NVFP4 has 6 of 8. Both PASS.InvalidInferenceResultError(streams without content), the same rate as in the source runs.max_running_requests: 48.max_running_requests: 96. CPU-tier cache hits are 0.4–0.8% at c64 and 3.1–4.3% at c96.Deviations from the cookbook (HiCache points only)
1. Auto-sized HiCache pools (fraction 0.76) instead of
--hicache-size 32--enable-hierarchical-cache --hicache-host-memory-fraction 0.76. Write policy, IO backend and layout stay at the defaults (write_through / kernel / page_first).2.
--max-running-requests 96on the HiCache pointsCPU DRAM accounting
The MODELS.md policy allows HGX B300 at most 3 TB per server with 8/8 GPUs.
How SGLang sizes the pools
Measured allocation
Total of the three "Allocating … host memory" lines (KV + Mamba + DSA indexer) across 8 ranks, from each HiCache server log:
total-cpu-dram-gb(dram-utilization)Why 0.76 and not the default 0.8
Pre-run expectations
The concurrency lists were chosen from the source runs on tok/s per GPU vs p90 E2EL and vs interactivity. The source runs are #4710 41301274173, #4711 41300767755, #4729 41352674113 and #4731 41386432269.
Expected E2EL frontier counts were 6 (FP8) and 7 (NVFP4).
How the measured points compare:
Settings outside the cookbook that are harness-only or default-equal
served-model-name,watchdog-timeout 1800,enable-metrics,PYTHONNOUSERSITE=1,SGLANG_TIMEOUT_KEEP_ALIVE=1800,SGLANG_CACHE_DIR, theAIPERF_*client settings, the NVFP4 checkpoint revision pin, and the B300runners.yamlmodel mappings for the staged checkpoints.SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0andSGLANG_GLM_NEXTN_MOE_PTPC=0. Both areEnvBool(False)in the v0.5.22 environ.py. The draft head is served as shipped, through the image's checkpoint-default path.mem-fraction-static,max-running-requestson the no-offload points,cuda-graph-max-bs-decode,chunked-prefill-size/max-prefill-tokensTORCH_CUDA_ARCH_LIST,trust-remote-code(noauto_mapin any checkpoint config)context-length(equal to the checkpoint's 1,048,576),expert-parallel-size 1model-loader-extra-config,enable-cache-reporthicache-*policy flags, which equal their defaults中文
本 PR 在单台 8x B300 上同时添加 GLM-5.3-Flash FP8 与 NVFP4 的 AgentX 配置：
modelopt_fp4。glm-5.3-flash__mtp__native__default-default黄金曲线（MTP5 = 3.60）。仅 HiCache 点偏离 cookbook：
--hicache-host-memory-fraction 0.76，而非--hicache-size 32。--max-running-requests 96。CPU DRAM 计算：
实测结果（sweep 42024862281，全部通过，所有点
power_valid=1）：HiCache 实测主机内存分配：
dram-utilization改为 0.935 / 0.939，记录值为 2804 / 2816 GB。AI model disclosure
claude-opus-5-5):