Repository navigation
Append B300 GLM-5.3-Flash cache points / 追加 B300 GLM-5.3-Flash 缓存测试点 - #4912
functionstackx wants to merge 4 commits into
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
functionstackx
added
the
full-sweep-fail-fast
Perplexity Teammate
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Apart from the inline finding, I checked whether the existing points stay isolated, as append-only: true requires. They do: the &tp4_tuned_args anchor on override_none does not change what that block renders, the existing TP8 rows still point at an unchanged override_dram, and the <<: merge key follows a pattern other srt-slurm recipes in this repo already use.
Extended reasoning...
This PR changes three config files: one srt-slurm recipe, nvidia-master.yaml and perf-changelog.yaml. It touches no security-sensitive code. One finding about the TP4 HiCache host-memory fraction is posted inline. The anchor and merge-key change leaves the rendered recipes for the existing TP4 none and TP8 dram points unchanged.
| override_tp4_dram: | ||
| roles: | ||
| agg: | ||
| gpus: 4 | ||
| args: | ||
| <<: *tp4_tuned_args | ||
| enable-hierarchical-cache: true | ||
| hicache-host-memory-fraction: 0.76 |
There was a problem hiding this comment.
🔴 The new TP4 HiCache points may record only half the host DRAM their server actually uses. If so, the published per-GPU DRAM budget would be wrong. hicache-host-memory-fraction: 0.76 was sized for TP8 to fill the whole node, about 2.8 TB per the PR #4710 changelog entry. If that fraction is applied to the node's free host memory, a TP4 server on an exclusive --mem=0 node gets the same ~2.8 TB. The matrix records only the 4/8 share, about 1402 GB. Fix: size the TP4 pool to its 4/8 GPU share, e.g. with a smaller fraction or an explicit host size. dram-utilization is capped at 1, so the metadata cannot be raised to match a full-node pool.
Why this was flagged
This holds only if SGLang sizes the HiCache host pool from the node's free host memory, whatever the TP size. The comments at agentic.yaml:80-81 and the PR #4710 changelog ("host-memory fraction 0.76, about 2.8 TB per node, within the 3 TB cap") describe it that way. override_tp4_dram (agentic.yaml:99-109) reuses 0.76 for a TP4 server. b300-dsxe runs with slurm exclusive: true and salloc --mem=0 (runners.yaml:545-548), so per CONFIGS.md:223-225 the job can see the whole node's memory. agentic_dram_offload_gb (infx/matrix/generate.py:492-497) gives the TP4 row total-cpu-dram-gb = 2,861,022 MiB x 0.935 x 4/8, about 1402 GB. results/agentic/init.py:205 publishes that as allocated_cpu_dram_gb. If the pool is ~2.8 TB, the TP4 HiCache points (conc 52 and 64, nvidia-master.yaml:7573) get about twice their GPU share of host KV. That is published as 1402 GB and compared on the same offload curve as TP8. On the base branch no TP4 HiCache point exists. The validator at validation.py:837-838 caps dram-utilization at 1, so the metadata cannot record more than ~1500 GB for TP4.
Verification: override_tp4_dram (agentic.yaml:99-109) sets gpus: 4 and hicache-host-memory-fraction: 0.76. b300-dsxe sets exclusive: true and salloc-args: [--mem=0] (runners.yaml:545-548). Per CONFIGS.md:223-225, the job therefore sees all of the node's memory. The matrix publishes 1402 GB, about half the real figure, and the server exceeds the budget the config says it may use.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42432241904 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42432241904 |
Add four B300 GLM-5.3-Flash FP8 AgentX points to the existing curve:
Keep every existing point and recipe unchanged. The new TP8 cache-OFF variant uses the same server admission cap as TP8 HiCache. The new TP4 HiCache variant retains the tuned TP4 scheduling, memory split, and 32-request server admission cap, then enables the host cache.
TP4 HiCache uses host-memory fraction
0.50of available host memory. The existingdram-utilization: 0.935and GPU-proportional DRAM reporting are unchanged; actual host-pool allocation still needs runtime verification against the reported budget.The changelog uses
append-only: true, so the fail-fast sweep runs only these four new points and preserves the published curve. Append-only sweeps do not run fresh accuracy evals and cannot use theall-evalsmodifier. TP4 still reserves the whole eight-GPU node.中文
为现有 B300 GLM-5.3-Flash FP8 AgentX 曲线增加四个测试点：TP8 在客户端并发 52 时分别关闭和开启 HiCache，TP4 在客户端并发 52 和 64 时开启 HiCache。
保留所有现有测试点及其配置。新增 TP8 关闭缓存配置采用与 TP8 HiCache 相同的服务端请求上限；新增 TP4 HiCache 配置保留已调优的 TP4 调度、显存分配和 32 请求上限，再启用主机缓存。
TP4 HiCache 使用
0.50的可用主机内存比例。现有dram-utilization: 0.935和按 GPU 数量分配的 DRAM 报告保持不变；实际主机缓存池分配仍需在运行时核对是否符合报告预算。changelog 使用
append-only: true，fail-fast sweep 只运行这四个新增点，并保留已发布曲线。append-only sweep 不运行新的准确率评测，也不能使用all-evals修饰标签。TP4 仍独占整个八卡节点。AI model disclosure
GPT-6 Astra prepared the configuration changes and checked the append-only matrix and generated recipes.
Created with Perplexity