Skip to content

[DNM, Experimental] Add GLM-5.3-Flash NVFP4 B300 AgentX / [DNM, Experimental] 添加 GLM-5.3-Flash NVFP4 B300 AgentX - #4711

Closed
functionstackx wants to merge 10 commits into
mainfrom
feat/glm53flash-fp4-b300-sglang
Closed

functionstackx wants to merge 10 commits into
mainfrom
feat/glm53flash-fp4-b300-sglang

Conversation

@functionstackx

@functionstackx functionstackx commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Add GLM-5.3-Flash NVFP4 AgentX coverage on B300 with TP8, EAGLE/MTP 5/1/6 and concurrency 1, 2, 4, 8, 16, 32. The recipe uses the official digest-pinned SGLang v0.5.22-cu130 image, TRT-LLM DSA prefill/decode, FP8 KV and FlashInfer TRT-LLM MoE from the pinned upstream NVIDIA NVFP4 cookbook.

This is the Flash hybrid MLA/DSA/KDA model, with the full 1,048,576-token context and unfiltered AgentX corpus. It does not change the separate GLM-5.3 configuration, patch the engine, download weights, or collect another acceptance curve. The shared connector selects the MTP5 thinking-on point from #4708; that common model curve was measured with FP8 vLLM in the collector introduced by #4701.

The model is nvidia/GLM-5.3-Flash-NVFP4 at revision 1d00738646cbb0c6d0a9eb403dac5f366f1473a7, as reported by the checkpoint metadata. Its quantization exclusions cover embedded draft layer 45; the recipe leaves draft loading to the pinned upstream implementation and explicitly sets SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0 and SGLANG_GLM_NEXTN_MOE_PTPC=0.

Draft precision qualification: the runtime fingerprint identifies build baa9befa381511cd4d56f83980951e5a45fffc6d, and the exact-build inherited NextN constructor clears the modelopt_fp4 decoder quantization configuration before decoder construction. Both conversion flags remain 0, and no draft quantization override is introduced. This exact-build source-path evidence and checkpoint metadata support draft-as-shipped loading, but do not directly observe live tensor dtypes; the global NextN wrapper's NVFP4 log is not a decoder-precision measurement.

The successful runtime artifact confirms the staged /data/models/GLM-5.3-Flash-NVFP4:/model mount, full 1,048,576-token context, TP8/EP1, EAGLE 5/1/6, the declared model revision, and native AL 3.603536. No weight download was added.

Stack: this ready PR targets feat/glm53flash-golden-al (#4708), which depends on collector #4701. It is not intended to merge before those prerequisites.

Validation: all 102 targeted workload/synthetic-acceptance tests passed, and the full sweep for current head 6950741 completed successfully across c1/c2/c4/c8/c16/c32, independent c32 evaluation, and result collection. The regenerated current-head matrix matches all six collected recipe fingerprints (f3e422e1b1c5fe483c52f7dad4985b24ea80c74ba72cf34ee4af1c8881a9f4c4). Every point reports power_valid=1, the full 393-entry trace corpus, a 3,600-second profiling command without trace filtering, and zero context-overflow skips.

The result is not error-free. Direct phase-separated JSONL accounting finds 8 profiling errors out of 10,287 records (10,279 successful) and 447 warmup errors out of 695 records. Profiling errors by concurrency are 1/1/2/1/2/1; warmup errors are 11/15/34/51/105/231. All are empty-content InvalidInferenceResultError; the profiling errors affect three traces and their cause remains unresolved. Combined records_error_dropped is not a profiling-error count. The qualified audit comment records the per-point details.

The canonical GSM8K evaluation uses five shots, all 1,319 examples, and no sample limit. Strict accuracy is 1281/1319 (0.9711902956785443); flexible accuracy is 1280/1319 (0.9704321455648218). The evaluation runtime fingerprint contains no synthetic-acceptance variables and retains both prohibited conversion flags at 0.

中文

为 B300 添加 GLM-5.3-Flash NVFP4 AgentX 配置,使用 TP8、EAGLE/MTP 5/1/6,以及并发 1、2、4、8、16、32。官方 SGLang v0.5.22-cu130 镜像按 digest 固定,采用上游 cookbook 的 TRT-LLM DSA、FP8 KV 和 FlashInfer TRT-LLM MoE。

保留完整 1,048,576-token 上下文和未截断的 AgentX 数据集,不修改独立的 GLM-5.3 配置,不打引擎补丁,不下载权重,也不重新采集 AL。共享连接器读取 #4708 的模型公共 MTP 曲线;该曲线由 #4701 引入的 FP8 vLLM collector 实测。

模型为 nvidia/GLM-5.3-Flash-NVFP4,revision 为 1d00738646cbb0c6d0a9eb403dac5f366f1473a7。运行确认 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE 和 SGLANG_GLM_NEXTN_MOE_PTPC 均为 0,没有显式 draft 量化覆盖。精确 build baa9befa381511cd4d56f83980951e5a45fffc6d 的继承 NextN 构造函数在建立 decoder 前清除 modelopt_fp4 量化配置;该代码路径和 checkpoint 元数据支持原始 draft 加载,但不是对运行中 tensor dtype 的直接观测,也不能把全局 wrapper 的 NVFP4 日志当作 decoder 精度证明。

成功运行已确认 /data/models/GLM-5.3-Flash-NVFP4:/model 挂载、完整 1,048,576-token 上下文、TP8/EP1、EAGLE 5/1/6 和 AL 3.603536。当前 head 6950741 的六个并发点、独立 canonical eval 及结果收集均通过;当前矩阵和全部 artifact fingerprint 一致,所有点 power_valid=1,使用完整 393-entry 数据集、3600 秒 profiling,未过滤 trace,context-overflow skip 为零。

结果并非零错误:profiling 共 10,287 条记录中 8 条错误、10,279 条成功,warmup 695 条记录中 447 条错误。按并发顺序,profiling 错误为 1/1/2/1/2/1,warmup 错误为 11/15/34/51/105/231;均为空内容响应,profiling 涉及三条 trace,根因仍未解决。不能把合并阶段的 records_error_dropped 当作 profiling 错误数。

GSM8K 使用五样本、完整 1,319 题、无样本截断,strict 为 1281/1319,flexible 为 1280/1319;eval 进程不含 synthetic-acceptance 环境变量。PR 仍以 #4708 分支为目标并依赖 #4701,保持原 head 和唯一 full-sweep-fail-fast 标签;本次审计未重跑、restack 或合并,前置 PR 合并前不应合并。

AI model disclosure

GPT-6 Astra coordinated the four sibling PRs. GPT-6 Astra delegated agents implemented this B300 NVFP4 recipe and cross-checked checkpoint, image, and draft-precision evidence with the B200 NVFP4 agent.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@functionstackx functionstackx added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Comment on lines +8823 to +8835
glm5.3flash-fp4-b300-sglang-agentic-mtp:
image: lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae
model: nvidia/GLM-5.3-Flash-NVFP4
model-prefix: glm5.3flash
runner: cluster:b300-dsxe
precision: fp4
framework: sglang
multinode: false
srt-recipe-dir: glm5.3flash/sglang/b300-fp4-mtp
scenarios:
agentic-coding:
- search-space:
- { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32], srt-recipe: agentic.yaml }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Merging this adds a new benchmark that never gets results collected, because perf-changelog.yaml has no entry for it. AGENTS.md:87 says any change that alters what a benchmark runs needs a changelog entry for the affected config keys, since that entry triggers result collection. This PR adds the new config key glm5.3flash-fp4-b300-sglang-agentic-mtp and its recipe but leaves inferencex-e2e/perf-changelog.yaml untouched. Fix: append one English-only block at the tail of inferencex-e2e/perf-changelog.yaml. It should list config-keys glm5.3flash-fp4-b300-sglang-agentic-mtp and scenario-type agentic-coding, and keep every existing byte unchanged.

Why this was flagged

The diff adds the master entry glm5.3flash-fp4-b300-sglang-agentic-mtp at nvidia-master.yaml:8823-8835 and the recipe glm5.3flash/sglang/b300-fp4-mtp/agentic.yaml. Both define a benchmark that does not exist on the base branch. A grep of inferencex-e2e/perf-changelog.yaml finds no glm5.3flash key. The diff does not touch that file at all. AGENTS.md:87 says the changelog entry is what triggers result collection for affected config keys. Without it, this key's AgentX sweep (conc 1-32) is never scheduled, so no results appear after merge. The sibling glm5.3-fp4-b300-sglang-agentic-mtp has its own changelog block at perf-changelog.yaml:10869-10870, which shows the expected pattern.

Verification: The diff adds config key glm5.3flash-fp4-b300-sglang-agentic-mtp, but inferencex-e2e/perf-changelog.yaml is unchanged and has no glm5.3flash entry. AGENTS.md:87 says: "A change that alters what any benchmark runs needs a perf-changelog.yaml entry for the affected config keys; the entry triggers result collection." So the new conc 1-32 AgentX points will never be scheduled or collected.

@functionstackx functionstackx Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit 66bb1d6, after the reviewed initial commit 334faa1. The current head 6950741 includes exactly one append-only English changelog block for glm5.3flash-fp4-b300-sglang-agentic-mtp / agentic-coding, with this PR URL. The current-head changelog gate passed and full sweep is queued/running; GPU results are not complete yet.

中文:该问题针对初始提交 334faa1,已在后续提交 66bb1d6 修复。当前 head 6950741 包含一个仅追加的英文 changelog 条目,列出准确的 config key、agentic-coding 和本 PR 链接。当前 head 的 changelog 检查已通过,全量 sweep 已启动,等待 GPU 执行;尚未取得完整 GPU 结果。

@functionstackx functionstackx changed the title Add GLM-5.3-Flash NVFP4 B300 AgentX / 添加 GLM-5.3-Flash NVFP4 B300 AgentX [DNM, Experimental] Add GLM-5.3-Flash NVFP4 B300 AgentX / [DNM, Experimental] 添加 GLM-5.3-Flash NVFP4 B300 AgentX Nov 1, 2026
@functionstackx

Copy link
Copy Markdown
Collaborator Author

The full sweep passed on current head 6950741: all six concurrency points (1/2/4/8/16/32), the canonical c32 evaluation, and result collection succeeded. The regenerated current-head matrix matches all six result fingerprints; every point has power_valid=1, the full 393-entry trace corpus, and a 3,600-second profiling command.

This is qualified success, not error-free. Direct phase-separated JSONL counts show profiling errors 1/1/2/1/2/1 out of 305/456/700/1400/2663/4763 records, and warmup errors 11/15/34/51/105/231 out of 11/22/44/87/177/354 records. All are empty-content InvalidInferenceResultError; profiling errors affect three traces. No point records a context-overflow skip. Combined records_error_dropped must not be reported as profiling errors.

The canonical evaluation is GSM8K five-shot, 1,319/1,319 examples, no sample limit: strict accuracy 0.9711902956785443 and flexible accuracy 0.9704321455648218. The evaluation process fingerprint contains no synthetic-acceptance variables.

The runtime artifact confirms the staged /data/models/GLM-5.3-Flash-NVFP4 mount, context 1,048,576, TP8/EP1, EAGLE 5/1/6, AL 3.603536, and both conversion flags 0. The exact SGLang build is baa9befa381511cd4d56f83980951e5a45fffc6d; its inherited NextN constructor clears modelopt_fp4 for the decoder. This source-path evidence plus checkpoint metadata supports draft-as-shipped loading, but does not directly observe live tensor dtypes. The global NextN wrapper's NVFP4 log alone is not a decoder-precision measurement.

中文:当前 head 6950741 的全量 sweep、六个并发点及独立 canonical eval 均通过,结果 fingerprint 与当前矩阵一致。所有点 power_valid=1,保留完整 393 条 trace 数据集、3600 秒运行和 1,048,576-token 服务上下文。

本次结果并非零错误:按 JSONL 阶段拆分,profiling 错误数依次为 1/1/2/1/2/1,warmup 错误数为 11/15/34/51/105/231,均为空内容响应;profiling 错误涉及三条 trace,context-overflow skip 为零。GSM8K 使用五样本、完整 1319 题、无样本截断,strict/flexible 分别为 0.9711902956785443 / 0.9704321455648218,eval 不含 synthetic-acceptance 环境变量。运行日志及精确版本代码支持原始 draft 加载路径,但不是对运行中 tensor dtype 的直接观测。本次审计未重跑、restack 或合并 PR。

@Oseltamivir

Copy link
Copy Markdown
Collaborator

/use 41300767755

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@Oseltamivir staging run 41300767755. Existing staged runs will be preserved; a completion link will be posted here.

@Oseltamivir
Oseltamivir changed the base branch from feat/glm53flash-golden-al to main November 5, 2026 11:26
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +551 -4

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@Oseltamivir

Copy link
Copy Markdown
Collaborator

Superseded by #4710 (merged as 23a74c6), which consolidates GLM-5.3-Flash B300 FP8 + NVFP4 base and HiCache points into one PR (no TP4 on the 8-GPU node). / 已由 #4710(合并为 23a74c6)取代,该 PR 将 GLM-5.3-Flash B300 FP8 与 NVFP4 的基础点和 HiCache 点合并为一个 PR(8 卡节点不跑 TP4)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants