Repository navigation
[DNM, Experimental] GLM-5.3-Flash B300 FP8 HiCache/concurrency / [DNM, Experimental] GLM-5.3-Flash B300 FP8 HiCache/并发实验 - #4731
Closed
functionstackx wants to merge 15 commits into
Closed
functionstackx wants to merge 15 commits into
functionstackx wants to merge 15 commits into
Conversation
…erplexity.ai/SemiAnalysisAI/InferenceX into feat/glm53flash-fp8-b300-sglang
…erplexity.ai/SemiAnalysisAI/InferenceX into feat/glm53flash-fp8-b300-sglang
Contributor
|
Thanks for the contribution!
中文感谢你的贡献!
|
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41386432269 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41386432269 |
Collaborator
|
/use 41386432269 |
Contributor
|
@Oseltamivir staging run 41386432269. Existing staged runs will be preserved; a completion link will be posted here. |
Oseltamivir
changed the base branch from
feat/glm53flash-fp8-b300-sglang
to
main
November 5, 2026 11:26
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft child of #4710, based on its passed head
dcc3d233e1ff1b29b318c032f3c21a34a2db389a; the parent remains unchanged. The first authorized GPU sweep failed; one controlled retry after the scoped admission repair completed, with the qualifications below.Completed controlled retry
Run 41386432269 completed all eight performance points and the fresh canonical evaluation at exact head
ea39a81691d39a0383b4b642a71bbbeab3504da1. OFF48 finished at 06:00:20 UTC, the gate completed at 06:00:25, and remaining jobs started no earlier than 06:03:28; all eight recipe fingerprints match the corrected plan.Both c96 performance logs and the fresh canonical log confirm admission cap 96 and graph cap 96, with observed peak residency 96. The inspected c96 TP0 logs contain no OOM, error, or retraction; this successful retry is not a universal OOM cure, and c80 still has admission cap 160, making the 80/96 comparison explicitly admission-tuned rather than a pure concurrency/cache ablation (OFF96, ON96).
At matched c96, ON versus OFF achieved aggregate input-plus-output throughput 404,655.22832 versus 315,404.06088 TPS (+28.30%) and app P90 interactivity 35.932447 versus 21.649708 (+65.97%), with X computed as
1 / full_response_itl.p90. Across the combined original-native and child observations, the native frontier collectively dominates seven of the eight new points: native c48 dominates both new c48 points, native c64 dominates the other five non-ON96 points, and only ON96 adds a nondominated point; these are single-run observations, not replicated gains (native run, retry results).Raw profiling recorded 70,632 successes and 13 empty-content failures among 70,645 completed records, separately from 3,947 warmup errors and 32 request-level teardown cancellations outside JSONL. External cancellations by OFF/ON pair are 48=4/3, 64=9/6, 80=3/1, 96=5/1; their counts are not included in the completed-record denominator or mixed with session counters. The full 393 traces and 3,600-second commands remain intact, with maximum observed input 950,586 tokens, runtime context 1,048,576, and no context-overflow skips (raw run artifacts).
Fresh SIM-OFF canonical GSM8K strict and flexible scores both equal 1,281/1,319 (97.1190%). All ON performance points recorded host-cache activity, but those performance runs used synthetic acceptance; no forced simulation-OFF host-restore proof was collected, so the ordinary canonical pass is not host-restore certification (fresh canonical evaluation).
All eight raw power packages pass validation and package-integrity checks, with complete/publication-valid manifests and eight expected/observed GPUs; maximum sample gaps are 1.2605–1.2976 seconds. This covers eight GPU boards, excludes CPU/DRAM and whole-node energy, and does not alone establish an energy-efficiency gain (power artifacts).
Failed sweep and admission repair
Run 41353165110 at
75c88aae3f06995bef63269e79a7e6391531a977failed: ON96 hit CUDA OOM in EAGLE draft DSA page-tablerepeat_interleave, requesting 2.86 GiB with 618 MiB free. The last observed resident batch was 166 against admission cap 192 and graph cap 96; this is not evidence of host RAM exhaustion or a completed eight-point sweep.ON48 completed its benchmark and exported a formal row before its workflow job was cancelled by fail-fast. Seven formal rows exist, including that cancelled-job artifact; across those seven complete benchmark artifacts, profiling recorded 54,764 successes and six empty-content failures, separately from 3,321 warmup errors and 45 profiling teardown cancellations outside JSONL.
The ON96 canonical evaluation passed with simulation disabled: GSM8K strict 1,281/1,319 (97.1190%), flexible 1,278/1,319 (96.8916%). It does not validate the failed long-context performance point or establish forced simulation-OFF host-restore correctness.
Seven formal points have valid eight-GPU board-power packages and complete, integrity-valid telemetry; ON96 has an incomplete, publication-invalid package and no formal result. GPU board power is not whole-node CPU/DRAM energy, and valid partial artifacts do not turn this failed sweep into a full pass (run artifacts).
Repair
ea39a81691d39a0383b4b642a71bbbeab3504da1changes only matched OFF96/ON96max-running-requests: 192to96, retaining client concurrency 96, graph cap 96, all other points, full workload, image, precision, MTP, and HiCache fraction 0.65. The ON96 canonical alias inherits the cap and requires fresh evaluation; concurrency 80 retains admission cap 160, so this is explicitly admission tuning with a nonmonotonic cap schedule, not a pure concurrency/cache ablation or a guaranteed OOM cure.The actual eight-performance-plus-one-evaluation render changes only the two c96 performance fingerprints and three c96 bound commands; all other six performance payloads are unchanged. A generic regression tests explicit admission caps 4/8/16 independently of client/graph concurrency 8 in both performance and evaluation modes; 117 focused tests and CI-configured Ruff checks passed locally, with independent read-only review completed.
Experiment
Compare HiCache OFF and ON at concurrency 48, 64, 80, and 96, with matching request and graph caps in each pair. ON uses
enable-hierarchical-cache,hicache-host-memory-fraction: 0.65,write_through,kernel, andpage_first; it sets no explicithicache-sizeorhicache-ratio. OFF metadata iskv-offloading: none; ON isdramwith backendhicache, and both selectors bind the corresponding benchmark metadata.Keep the parent's TP8, FP8 target/KV configuration, checkpoint-default MTP precision, EAGLE 5/1/6, and official digest
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae(runtime sourcebaa9befa381511cd4d56f83980951e5a45fffc6d). Keep the full 393-entry AgentX corpus, 3,600-second profile, canonical warmup, and model-default 1,048,576-token context. The sharedglm5.3flashloader supplies configured synthetic MTP5 acceptance3.603536from the golden curve, not a new runtime measurement; the recipe contains no manual acceptance override or engine patch.The performance changelog entry disables inline evaluation. A separate ON-only master selector at concurrency 96 and an
evals-onlychangelog entry preserve a full canonical GSM8K evaluation without synthetic acceptance. The passed parent OFF evaluation is historical evidence, not a fresh child result.Initial local validation passed 121 focused tests and asserts exactly eight paired performance rows plus one ON concurrency-96 evaluation row. All nine bound recipes pass the real srtctl schema and production SGLang worker-command builder. Paired non-cache arguments and resource caps match, ON nominal DRAM metadata is 2,999 GB under the legacy generator cap, every performance row receives configured synthetic AL 3.603536, and the actual evaluation-bound role has no synthetic-acceptance environment. The production lm-eval function was locally captured with subprocess execution suppressed: it selects the canonical GSM8K YAML, concurrency 96, 1,048,576 context, and no
--limit. The full sbatch-render CLI still requires compute-runtime binaries absent from this sandbox; those local checks are distinct from the qualified GPU runs above.Why these points
The parent sweep increased output throughput from 2,044.25 to 2,414.84 tokens/s between concurrency 48 and 64, while P90 interactivity fell from 102.29 to 44.28 tokens/s. GPU cache-hit rate fell from 96.676% to 96.181%, and reported KV utilization rose from 39% to 66%. These measurements motivate matched anchors at 48/64 and bounded extensions at 80/96; improved performance is a hypothesis, not a result.
The concurrency-64 runtime reported eight 275,040-MiB GPUs, target weights 38.06 GiB/rank, NextN weights 1.21 GiB/rank, target KV 122.08 GiB/rank, draft KV 11.10 GiB/rank, and 27.65 GiB/rank free after graph capture. Its logical KV capacity is 21,864,704 tokens; the aggregate 174,917,632-token metric sums eight ranks and must not be treated as eight times the usable context capacity. Extrapolating the observed pool-size slope gives roughly 21.32M/20.78M logical tokens at concurrency 80/96; assuming linear growth of the cached working set gives roughly 85%/104% utilization. This estimate excludes changes in request mix and graph overhead and is not a capacity guarantee.
The B300 inventory declares 3,883 GiB eligible host RAM, exclusive allocation, and
--mem=0. That is not measured liveMemAvailableor NUMA headroom. Both arms preserve those existing Slurm resources. Matrixdram-utilization: 1.0means 100% of the existing nominal eligible budget, subject to the generator's legacy 3,000-GB cap where applicable; it is neither reserved RAM nor measured host-pool allocation. OFF's generated zero DRAM field is existing offload metadata semantics, not zero allocated host RAM. The separate engine fraction 0.65 controls live automatic sizing, and actual resolved pool bytes must be captured at runtime.Remaining qualification
The pinned hybrid allocator handles target, draft, KDA, and indexer pools, but upstream #42918 documents why a fixed host-size value is not a reliable total physical-allocation cap. The retry captured live host/cgroup headroom and native autosizing evidence, but forced host restoration with synthetic acceptance disabled remains unproven. Merely enabling HiCache or passing the ordinary canonical ON evaluation is not host-restore proof.
The parent is not error-free: its raw records contain 10 profiling empty-content errors among 29,599 requests, separate from 1,355 warmup errors. Their cause remains unresolved; this child does not shorten the workload or claim HiCache fixes them.
中文
本草稿 PR 直接基于 #4710 已通过的提交
dcc3d233e1ff1b29b318c032f3c21a34a2db389a,不修改父分支。首轮授权 GPU sweep 失败;窄范围 admission 修复后的一次受控重试已完成,以下保留结果限制及原失败历史。受控重试 run 41386432269 在 exact head
ea39a81691d39a0383b4b642a71bbbeab3504da1完成全部八个性能点及新的标准评测。OFF48 于 UTC 06:00:20 完成,gate 于 06:00:25 完成,其余任务最早 06:03:28 开始;八个 recipe 指纹均匹配修复后的计划。两组 c96 性能日志及新标准评测均确认 admission cap=96、graph cap=96,实际峰值 resident requests=96;检查的 c96 TP0 日志没有 OOM/error/retraction。这不代表通用 OOM 治愈,且 c80 仍是 cap160,因此 80/96 是 admission 调优比较,不是纯并发/cache 消融(OFF96、ON96)。
相同 c96 下,ON/OFF 输入加输出总吞吐分别为 404,655.22832 / 315,404.06088 TPS(+28.30%),应用 P90 交互性为 35.932447 / 21.649708(+65.97%);X 使用
1 / full_response_itl.p90。与原生父运行合并后,原生前沿整体支配七个新点:原生 c48 支配两个新 c48 点,原生 c64 支配其余五个非 ON96 点,仅 ON96 新增非支配点。这是单次观察,不是重复实验确认的收益(原生运行、重试结果)。原始 profiling 的 70,645 条完成记录含 70,632 次成功、13 次空内容失败,另有 3,947 次 warmup 错误及 JSONL 之外的 32 次请求级 teardown cancellations。OFF/ON 分别为 48=4/3、64=9/6、80=3/1、96=5/1,不纳入完成记录分母,也不与 session 计数混合;完整 393 条数据和 3,600 秒命令不变,最大观察输入 950,586 tokens、运行上下文 1,048,576,无 context-overflow skip(原始产物)。
新的 SIM-OFF 标准 GSM8K strict/flexible 均为 1,281/1,319(97.1190%)。ON 性能点虽均有 host-cache hits,但性能运行使用合成接受率;未采集强制 SIM-OFF host restore 证明,普通标准评测通过不等于主机恢复认证(新标准评测)。
八点原始 power 数据包及完整性检查均通过,manifest 完整且可发布,expected/observed 均为八 GPU,最大采样间隔 1.2605–1.2976 秒。覆盖仅限八张 GPU board,不包括 CPU/DRAM 或整机能耗,也不能单独证明能效收益(power 产物)。
Run 41353165110 在提交
75c88aae3f06995bef63269e79a7e6391531a977的 ON96 点发生 CUDA OOM:EAGLE draft DSA page-tablerepeat_interleave需要 2.86 GiB,但只剩 618 MiB。最后观察到 166 个 resident requests,admission cap 为 192、graph cap 为 96;不能将此解释为主机 RAM 耗尽或八点 sweep 完整通过。ON48 已完成 benchmark 并生成正式结果,之后 workflow job 被 fail-fast 取消,必须保留 cancelled 状态。七个已完成 benchmark 的正式记录共 54,764 次 profiling 成功、6 次空内容失败,另有 3,321 次 warmup 错误及 JSONL 之外的 45 次 profiling teardown cancellations(运行产物)。
ON96 的标准 SIM-OFF GSM8K 为 strict 1,281/1,319(97.1190%)、flexible 1,278/1,319(96.8916%),但这不验证失败的长上下文性能点,也不构成强制 SIM-OFF 主机恢复证明。七点八卡 board-power 数据包有效;失败 ON96 的数据包不完整、不可发布,而且 GPU board power 不含整机 CPU/DRAM 能耗(评测及产物)。
修复
ea39a81691d39a0383b4b642a71bbbeab3504da1只将成对 OFF96/ON96 的max-running-requests从 192 改为 96,客户端并发、graph cap 及完整工作负载不变,其他点及镜像/精度/MTP/0.65 主机比例均不变。ON96 标准评测 alias 同样继承新 cap,必须重新评测;并发 80 仍保留 cap 160,因此这是 admission 调优及非单调 cap 计划,不是纯并发/cache 消融,也不保证消除 OOM。真实八性能点加一评测点的深度对比仅改变两个 c96 性能指纹及三个 c96 绑定命令,其余六点完全一致。通用回归在客户端和 graph 并发 8 下分别测试 admission cap 4/8/16 的性能/评测模式;本地 117 项测试及 CI Ruff 检查通过,并已完成独立只读评审。
新增并发 48/64/80/96 的 HiCache OFF/ON 成对配置,每对使用相同请求上限与 CUDA graph 上限。ON 使用自动主机内存比例 0.65、
write_through、kernel、page_first,不设置固定hicache-size或hicache-ratio;OFF/ON 的 offload 元数据分别为none和dram/hicache。两组保留父配置相同的实际 Slurm 资源。
dram-utilization: 1.0仅表示现有名义主机预算,仍遵循生成器已有的 3,000-GB 上限(如适用),不是实际预留内存或缓存分配量;OFF 的零值也不表示没有分配主机内存。引擎实际使用独立的 0.65 自动分配比例,运行时仍需记录真实缓存池字节数。保留原镜像摘要、TP8、FP8、checkpoint 默认 MTP 精度、EAGLE 5/1/6、完整 393 条 AgentX 数据、3,600 秒正式运行、标准预热及模型默认 1,048,576-token 上下文。性能点通过共享 loader 使用 golden curve 配置的合成接受长度 3.603536,并非本次运行的新实测值;独立的 ON 并发 96 selector 保留完整标准 GSM8K 评测,并移除合成接受率。
父 sweep 在并发 48 到 64 时输出吞吐从 2,044.25 增至 2,414.84 tokens/s,但 P90 交互速度从 102.29 降至 44.28 tokens/s。因此保留 48/64 对照,有限扩展至 80/96;原失败 sweep 的部分结果不作为完整通过依据,当前依据是上述有明确限制的受控重试。
剩余资格验证仍包括关闭合成接受率时的强制主机缓存恢复证明,普通标准 ON 评测不能填补该缺口;重试已捕获 live 主机/cgroup headroom 和引擎自动分配结果。父运行的 10 次正式阶段空内容错误和 1,355 次预热错误仍未解决,本 PR 不通过缩短工作负载掩盖这些错误,也不宣称 HiCache 已修复它们。
本地 121 项测试通过,实际生成计划严格为八个性能点和一个 ON 并发 96 评测点;九个绑定配置均通过真实 schema 和 SGLang 命令构建器验证。标准 GSM8K 命令在禁止子进程执行的条件下完成本地捕获,未含
--limit,评测服务环境不含合成接受率。完整 sbatch CLI 仍需要本地缺失的计算节点运行时二进制文件;这些验证不代表已完成 GPU 运行、主机缓存恢复或精度验证。AI model disclosure
GPT-6 Astra coordinated the research and implementation. A delegated GPT-6 Astra agent implemented and locally validated this B300 FP8 child.