Skip to content

[DNM, Experimental] GLM-5.3-Flash B300 FP8 HiCache/concurrency / [DNM, Experimental] GLM-5.3-Flash B300 FP8 HiCache/并发实验 - #4731

Closed
functionstackx wants to merge 15 commits into
mainfrom
feat/glm53flash-fp8-b300-sglang-hicache-frontier
Closed

functionstackx wants to merge 15 commits into
mainfrom
feat/glm53flash-fp8-b300-sglang-hicache-frontier

Conversation

@functionstackx

@functionstackx functionstackx commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Draft child of #4710, based on its passed head dcc3d233e1ff1b29b318c032f3c21a34a2db389a; the parent remains unchanged. The first authorized GPU sweep failed; one controlled retry after the scoped admission repair completed, with the qualifications below.

Completed controlled retry

Run 41386432269 completed all eight performance points and the fresh canonical evaluation at exact head ea39a81691d39a0383b4b642a71bbbeab3504da1. OFF48 finished at 06:00:20 UTC, the gate completed at 06:00:25, and remaining jobs started no earlier than 06:03:28; all eight recipe fingerprints match the corrected plan.

Both c96 performance logs and the fresh canonical log confirm admission cap 96 and graph cap 96, with observed peak residency 96. The inspected c96 TP0 logs contain no OOM, error, or retraction; this successful retry is not a universal OOM cure, and c80 still has admission cap 160, making the 80/96 comparison explicitly admission-tuned rather than a pure concurrency/cache ablation (OFF96, ON96).

At matched c96, ON versus OFF achieved aggregate input-plus-output throughput 404,655.22832 versus 315,404.06088 TPS (+28.30%) and app P90 interactivity 35.932447 versus 21.649708 (+65.97%), with X computed as 1 / full_response_itl.p90. Across the combined original-native and child observations, the native frontier collectively dominates seven of the eight new points: native c48 dominates both new c48 points, native c64 dominates the other five non-ON96 points, and only ON96 adds a nondominated point; these are single-run observations, not replicated gains (native run, retry results).

Raw profiling recorded 70,632 successes and 13 empty-content failures among 70,645 completed records, separately from 3,947 warmup errors and 32 request-level teardown cancellations outside JSONL. External cancellations by OFF/ON pair are 48=4/3, 64=9/6, 80=3/1, 96=5/1; their counts are not included in the completed-record denominator or mixed with session counters. The full 393 traces and 3,600-second commands remain intact, with maximum observed input 950,586 tokens, runtime context 1,048,576, and no context-overflow skips (raw run artifacts).

Fresh SIM-OFF canonical GSM8K strict and flexible scores both equal 1,281/1,319 (97.1190%). All ON performance points recorded host-cache activity, but those performance runs used synthetic acceptance; no forced simulation-OFF host-restore proof was collected, so the ordinary canonical pass is not host-restore certification (fresh canonical evaluation).

All eight raw power packages pass validation and package-integrity checks, with complete/publication-valid manifests and eight expected/observed GPUs; maximum sample gaps are 1.2605–1.2976 seconds. This covers eight GPU boards, excludes CPU/DRAM and whole-node energy, and does not alone establish an energy-efficiency gain (power artifacts).

Failed sweep and admission repair

Run 41353165110 at 75c88aae3f06995bef63269e79a7e6391531a977 failed: ON96 hit CUDA OOM in EAGLE draft DSA page-table repeat_interleave, requesting 2.86 GiB with 618 MiB free. The last observed resident batch was 166 against admission cap 192 and graph cap 96; this is not evidence of host RAM exhaustion or a completed eight-point sweep.

ON48 completed its benchmark and exported a formal row before its workflow job was cancelled by fail-fast. Seven formal rows exist, including that cancelled-job artifact; across those seven complete benchmark artifacts, profiling recorded 54,764 successes and six empty-content failures, separately from 3,321 warmup errors and 45 profiling teardown cancellations outside JSONL.

The ON96 canonical evaluation passed with simulation disabled: GSM8K strict 1,281/1,319 (97.1190%), flexible 1,278/1,319 (96.8916%). It does not validate the failed long-context performance point or establish forced simulation-OFF host-restore correctness.

Seven formal points have valid eight-GPU board-power packages and complete, integrity-valid telemetry; ON96 has an incomplete, publication-invalid package and no formal result. GPU board power is not whole-node CPU/DRAM energy, and valid partial artifacts do not turn this failed sweep into a full pass (run artifacts).

Repair ea39a81691d39a0383b4b642a71bbbeab3504da1 changes only matched OFF96/ON96 max-running-requests: 192 to 96, retaining client concurrency 96, graph cap 96, all other points, full workload, image, precision, MTP, and HiCache fraction 0.65. The ON96 canonical alias inherits the cap and requires fresh evaluation; concurrency 80 retains admission cap 160, so this is explicitly admission tuning with a nonmonotonic cap schedule, not a pure concurrency/cache ablation or a guaranteed OOM cure.

The actual eight-performance-plus-one-evaluation render changes only the two c96 performance fingerprints and three c96 bound commands; all other six performance payloads are unchanged. A generic regression tests explicit admission caps 4/8/16 independently of client/graph concurrency 8 in both performance and evaluation modes; 117 focused tests and CI-configured Ruff checks passed locally, with independent read-only review completed.

Experiment

Compare HiCache OFF and ON at concurrency 48, 64, 80, and 96, with matching request and graph caps in each pair. ON uses enable-hierarchical-cache, hicache-host-memory-fraction: 0.65, write_through, kernel, and page_first; it sets no explicit hicache-size or hicache-ratio. OFF metadata is kv-offloading: none; ON is dram with backend hicache, and both selectors bind the corresponding benchmark metadata.

Keep the parent's TP8, FP8 target/KV configuration, checkpoint-default MTP precision, EAGLE 5/1/6, and official digest lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae (runtime source baa9befa381511cd4d56f83980951e5a45fffc6d). Keep the full 393-entry AgentX corpus, 3,600-second profile, canonical warmup, and model-default 1,048,576-token context. The shared glm5.3flash loader supplies configured synthetic MTP5 acceptance 3.603536 from the golden curve, not a new runtime measurement; the recipe contains no manual acceptance override or engine patch.

The performance changelog entry disables inline evaluation. A separate ON-only master selector at concurrency 96 and an evals-only changelog entry preserve a full canonical GSM8K evaluation without synthetic acceptance. The passed parent OFF evaluation is historical evidence, not a fresh child result.

Initial local validation passed 121 focused tests and asserts exactly eight paired performance rows plus one ON concurrency-96 evaluation row. All nine bound recipes pass the real srtctl schema and production SGLang worker-command builder. Paired non-cache arguments and resource caps match, ON nominal DRAM metadata is 2,999 GB under the legacy generator cap, every performance row receives configured synthetic AL 3.603536, and the actual evaluation-bound role has no synthetic-acceptance environment. The production lm-eval function was locally captured with subprocess execution suppressed: it selects the canonical GSM8K YAML, concurrency 96, 1,048,576 context, and no --limit. The full sbatch-render CLI still requires compute-runtime binaries absent from this sandbox; those local checks are distinct from the qualified GPU runs above.

Why these points

The parent sweep increased output throughput from 2,044.25 to 2,414.84 tokens/s between concurrency 48 and 64, while P90 interactivity fell from 102.29 to 44.28 tokens/s. GPU cache-hit rate fell from 96.676% to 96.181%, and reported KV utilization rose from 39% to 66%. These measurements motivate matched anchors at 48/64 and bounded extensions at 80/96; improved performance is a hypothesis, not a result.

The concurrency-64 runtime reported eight 275,040-MiB GPUs, target weights 38.06 GiB/rank, NextN weights 1.21 GiB/rank, target KV 122.08 GiB/rank, draft KV 11.10 GiB/rank, and 27.65 GiB/rank free after graph capture. Its logical KV capacity is 21,864,704 tokens; the aggregate 174,917,632-token metric sums eight ranks and must not be treated as eight times the usable context capacity. Extrapolating the observed pool-size slope gives roughly 21.32M/20.78M logical tokens at concurrency 80/96; assuming linear growth of the cached working set gives roughly 85%/104% utilization. This estimate excludes changes in request mix and graph overhead and is not a capacity guarantee.

The B300 inventory declares 3,883 GiB eligible host RAM, exclusive allocation, and --mem=0. That is not measured live MemAvailable or NUMA headroom. Both arms preserve those existing Slurm resources. Matrix dram-utilization: 1.0 means 100% of the existing nominal eligible budget, subject to the generator's legacy 3,000-GB cap where applicable; it is neither reserved RAM nor measured host-pool allocation. OFF's generated zero DRAM field is existing offload metadata semantics, not zero allocated host RAM. The separate engine fraction 0.65 controls live automatic sizing, and actual resolved pool bytes must be captured at runtime.

Remaining qualification

The pinned hybrid allocator handles target, draft, KDA, and indexer pools, but upstream #42918 documents why a fixed host-size value is not a reliable total physical-allocation cap. The retry captured live host/cgroup headroom and native autosizing evidence, but forced host restoration with synthetic acceptance disabled remains unproven. Merely enabling HiCache or passing the ordinary canonical ON evaluation is not host-restore proof.

The parent is not error-free: its raw records contain 10 profiling empty-content errors among 29,599 requests, separate from 1,355 warmup errors. Their cause remains unresolved; this child does not shorten the workload or claim HiCache fixes them.

中文

本草稿 PR 直接基于 #4710 已通过的提交 dcc3d233e1ff1b29b318c032f3c21a34a2db389a,不修改父分支。首轮授权 GPU sweep 失败;窄范围 admission 修复后的一次受控重试已完成,以下保留结果限制及原失败历史。

受控重试 run 41386432269 在 exact head ea39a81691d39a0383b4b642a71bbbeab3504da1 完成全部八个性能点及新的标准评测。OFF48 于 UTC 06:00:20 完成,gate 于 06:00:25 完成,其余任务最早 06:03:28 开始;八个 recipe 指纹均匹配修复后的计划。

两组 c96 性能日志及新标准评测均确认 admission cap=96、graph cap=96,实际峰值 resident requests=96;检查的 c96 TP0 日志没有 OOM/error/retraction。这不代表通用 OOM 治愈,且 c80 仍是 cap160,因此 80/96 是 admission 调优比较,不是纯并发/cache 消融(OFF96、ON96)。

相同 c96 下,ON/OFF 输入加输出总吞吐分别为 404,655.22832 / 315,404.06088 TPS(+28.30%),应用 P90 交互性为 35.932447 / 21.649708(+65.97%);X 使用 1 / full_response_itl.p90。与原生父运行合并后,原生前沿整体支配七个新点:原生 c48 支配两个新 c48 点,原生 c64 支配其余五个非 ON96 点,仅 ON96 新增非支配点。这是单次观察,不是重复实验确认的收益(原生运行、重试结果)。

原始 profiling 的 70,645 条完成记录含 70,632 次成功、13 次空内容失败,另有 3,947 次 warmup 错误及 JSONL 之外的 32 次请求级 teardown cancellations。OFF/ON 分别为 48=4/3、64=9/6、80=3/1、96=5/1,不纳入完成记录分母,也不与 session 计数混合;完整 393 条数据和 3,600 秒命令不变,最大观察输入 950,586 tokens、运行上下文 1,048,576,无 context-overflow skip(原始产物)。

新的 SIM-OFF 标准 GSM8K strict/flexible 均为 1,281/1,319(97.1190%)。ON 性能点虽均有 host-cache hits,但性能运行使用合成接受率;未采集强制 SIM-OFF host restore 证明,普通标准评测通过不等于主机恢复认证(新标准评测)。

八点原始 power 数据包及完整性检查均通过,manifest 完整且可发布,expected/observed 均为八 GPU,最大采样间隔 1.2605–1.2976 秒。覆盖仅限八张 GPU board,不包括 CPU/DRAM 或整机能耗,也不能单独证明能效收益(power 产物)。

Run 41353165110 在提交 75c88aae3f06995bef63269e79a7e6391531a977 的 ON96 点发生 CUDA OOM:EAGLE draft DSA page-table repeat_interleave 需要 2.86 GiB,但只剩 618 MiB。最后观察到 166 个 resident requests,admission cap 为 192、graph cap 为 96;不能将此解释为主机 RAM 耗尽或八点 sweep 完整通过。

ON48 已完成 benchmark 并生成正式结果,之后 workflow job 被 fail-fast 取消,必须保留 cancelled 状态。七个已完成 benchmark 的正式记录共 54,764 次 profiling 成功、6 次空内容失败,另有 3,321 次 warmup 错误及 JSONL 之外的 45 次 profiling teardown cancellations(运行产物)。

ON96 的标准 SIM-OFF GSM8K 为 strict 1,281/1,319(97.1190%)、flexible 1,278/1,319(96.8916%),但这不验证失败的长上下文性能点,也不构成强制 SIM-OFF 主机恢复证明。七点八卡 board-power 数据包有效;失败 ON96 的数据包不完整、不可发布,而且 GPU board power 不含整机 CPU/DRAM 能耗(评测及产物)。

修复 ea39a81691d39a0383b4b642a71bbbeab3504da1 只将成对 OFF96/ON96 的 max-running-requests 从 192 改为 96,客户端并发、graph cap 及完整工作负载不变,其他点及镜像/精度/MTP/0.65 主机比例均不变。ON96 标准评测 alias 同样继承新 cap,必须重新评测;并发 80 仍保留 cap 160,因此这是 admission 调优及非单调 cap 计划,不是纯并发/cache 消融,也不保证消除 OOM。

真实八性能点加一评测点的深度对比仅改变两个 c96 性能指纹及三个 c96 绑定命令,其余六点完全一致。通用回归在客户端和 graph 并发 8 下分别测试 admission cap 4/8/16 的性能/评测模式;本地 117 项测试及 CI Ruff 检查通过,并已完成独立只读评审。

新增并发 48/64/80/96 的 HiCache OFF/ON 成对配置,每对使用相同请求上限与 CUDA graph 上限。ON 使用自动主机内存比例 0.65、write_through、kernel、page_first,不设置固定 hicache-size 或 hicache-ratio;OFF/ON 的 offload 元数据分别为 none 和 dram/hicache。

两组保留父配置相同的实际 Slurm 资源。dram-utilization: 1.0 仅表示现有名义主机预算,仍遵循生成器已有的 3,000-GB 上限(如适用),不是实际预留内存或缓存分配量;OFF 的零值也不表示没有分配主机内存。引擎实际使用独立的 0.65 自动分配比例,运行时仍需记录真实缓存池字节数。

保留原镜像摘要、TP8、FP8、checkpoint 默认 MTP 精度、EAGLE 5/1/6、完整 393 条 AgentX 数据、3,600 秒正式运行、标准预热及模型默认 1,048,576-token 上下文。性能点通过共享 loader 使用 golden curve 配置的合成接受长度 3.603536,并非本次运行的新实测值;独立的 ON 并发 96 selector 保留完整标准 GSM8K 评测,并移除合成接受率。

父 sweep 在并发 48 到 64 时输出吞吐从 2,044.25 增至 2,414.84 tokens/s,但 P90 交互速度从 102.29 降至 44.28 tokens/s。因此保留 48/64 对照,有限扩展至 80/96;原失败 sweep 的部分结果不作为完整通过依据,当前依据是上述有明确限制的受控重试。

剩余资格验证仍包括关闭合成接受率时的强制主机缓存恢复证明,普通标准 ON 评测不能填补该缺口;重试已捕获 live 主机/cgroup headroom 和引擎自动分配结果。父运行的 10 次正式阶段空内容错误和 1,355 次预热错误仍未解决,本 PR 不通过缩短工作负载掩盖这些错误,也不宣称 HiCache 已修复它们。

本地 121 项测试通过,实际生成计划严格为八个性能点和一个 ON 并发 96 评测点;九个绑定配置均通过真实 schema 和 SGLang 命令构建器验证。标准 GSM8K 命令在禁止子进程执行的条件下完成本地捕获,未含 --limit,评测服务环境不含合成接受率。完整 sbatch CLI 仍需要本地缺失的计算节点运行时二进制文件;这些验证不代表已完成 GPU 运行、主机缓存恢复或精度验证。

AI model disclosure

GPT-6 Astra coordinated the research and implementation. A delegated GPT-6 Astra agent implemented and locally validated this B300 FP8 child.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@functionstackx functionstackx changed the title [DNM, Experimental] B300 FP8 HiCache frontier / [DNM, Experimental] B300 FP8 HiCache 前沿实验 [DNM, Experimental] GLM-5.3-Flash B300 FP8 HiCache/concurrency / [DNM, Experimental] GLM-5.3-Flash B300 FP8 HiCache/并发实验 Nov 1, 2026
@functionstackx functionstackx added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

@functionstackx functionstackx removed the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 2, 2026
@functionstackx functionstackx added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 2, 2026
@Oseltamivir

Copy link
Copy Markdown
Collaborator

/use 41386432269

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@Oseltamivir staging run 41386432269. Existing staged runs will be preserved; a completion link will be posted here.

@Oseltamivir

Copy link
Copy Markdown
Collaborator

Superseded by #4710 (merged as 23a74c6), which consolidates GLM-5.3-Flash B300 FP8 + NVFP4 base and HiCache points into one PR (no TP4 on the 8-GPU node). / 已由 #4710(合并为 23a74c6)取代,该 PR 将 GLM-5.3-Flash B300 FP8 与 NVFP4 的基础点和 HiCache 点合并为一个 PR(8 卡节点不跑 TP4)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants