[functionstackx]

Draft child of #4731, based on 4a3c3e8deb4bbca6c29b5dae16e2cb500bd0938e. This is a TP4 plus admission-tuned configuration, not a pure TP-only causal comparison: every new point uses a common 96-request admission cap and decode CUDA-graph cap of 96. Existing parent recipes, branches, and results remain unchanged.

Experiment matrix

• Performance: HiCache OFF concurrency 8 as an anchor, then paired OFF/ON concurrency 48/64/80/96, nine performance points total.
• Accuracy: Fresh canonical OFF concurrency 16 and ON concurrency 96 evaluations, selected through separate master keys with evals-only and all-evals. Both retain full GSM8K and disable synthetic acceptance; different concurrencies prevent downstream evaluation-identity collisions.
• Serving: TP4/EP1, four active B300 GPUs within the unchanged exclusive eight-GPU reservation. Keep the parent official v0.5.22-cu130 image digest e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae, runtime source baa9befa381511cd4d56f83980951e5a45fffc6d, FP8 settings, checkpoint-default MTP precision, and EAGLE 5/1/6.
• Workload: Full 393-entry AgentX corpus, canonical warmup, 3,600-second profile, and unchanged model-default 1,048,576-token context. The shared loader selects measured MTP5 AL 3.603536 through glm5.3flash; there is no recipe acceptance override.
• HiCache: ON uses automatic host fraction 0.65, write_through, kernel, and page_first, without explicit size or ratio. OFF/ON have explicit none versus dram/hicache identities and otherwise matching serving caps.

The user-authorized full-sweep-fail-fast run has completed; this PR remains an experimental draft. Passing the OFF anchor and ordinary canonical evaluations does not prove simulation-OFF forced host restoration.

Completed sweep, qualified results

Run 41352868379 completed successfully at exact head 82c2ac3e3594cf8b12ca3a2399631dba34782d2a, with all nine performance points and both canonical evaluations. The OFF8 canary finished before the gate and fanout; all nine artifact fingerprints match the exact-head plan, retaining the full 393-entry dataset, 3,600-second commands, and 1,048,576-token runtime context.

The performance results show aggregate OFF/ON output TPS of 1,568.15/1,829.03 at concurrency 48, 1,358.54/2,298.98 at 64, 1,040.44/2,436.90 at 80, and 484.23/745.50 at 96. ON80 had the highest sampled output-token throughput only; ON96 degraded despite completing, and its slowdown mechanism remains unproven.

For the total-token TCO view, aggregate input-plus-output TPS for ON48/64/80/96 is 271,093.61018 / 306,458.15223 / 287,825.17117 / 98,620.97659. ON64, not ON80, maximizes total-token throughput, and ON64 also has higher app P90 interactivity, 43.898156 versus 26.075619, so it dominates ON80 on those two axes; X is 1 / full_response_itl.p90, not the raw exported interactivity field. ON64 total-token throughput is 76,614.53806 per active GPU or 38,307.26903 per reserved GPU before price normalization (total-token results).

The full canonical evaluations, both with simulation disabled, returned GSM8K strict 1,284/1,319 (97.3465%) and flexible 1,283/1,319 for OFF16, versus strict 1,281/1,319 (97.1190%) and flexible 1,280/1,319 for ON96. Different concurrencies and small count differences do not establish a paired statistical accuracy effect.

Raw profiling records contain 45,942 successes and 7 failures among 45,949 requests, separately from 4,049 warmup errors; all seven profiling failures are recurring empty-content InvalidInferenceResultError, whose underlying cause remains unresolved (run artifacts). No raw records carried a context-overflow skip, and the largest observed input was 847,410 tokens.

Profiling phase summaries additionally report 298 cancelled requests outside JSONL: OFF8=0, OFF48=1, ON48=0, OFF64=12, ON64=5, OFF80=44, ON80=7, OFF96=109, ON96=120. These are the request-level cancelled= values in the raw Phase profiling ... complete lines, not their separately logged session counters; the 45,949 completed-record denominator above does not include these unfinished requests (benchmark logs).

Actual B300 startup confirms 924 KDA slots, 15,815,936 logical GPU KV tokens, 75.45/2.28 GiB main/NextN weights per rank, and approximately 25 GiB GPU memory after graph capture. Native ON auto-sizing reports approximately 625 GiB/rank host budget, an estimated 299.3 GiB/rank pool footprint, and default ratio 2 retained without a recipe ratio override; separately logged host allocations are 194.35 GB KV, 70.24 GB Mamba, and 12.53 GB indexer per rank, using runtime display units (ON80 runtime).

All ON performance points recorded host-cache hits, but these were synthetic-acceptance performance runs. No forced simulation-OFF host-restore proof was collected, so the ordinary canonical passes are not full host-restore qualification (performance, accuracy).

Resource snapshots confirm four active GPUs within eight reserved GPUs. All nine power_valid results pass for four expected/observed GPUs only; they do not cover idle reserved GPUs or whole-node CPU/DRAM energy, and full-reserved-node cost-normalized throughput remains half the four-active-GPU view at the same per-GPU cost assumption (run artifacts).

TP4 capacity evidence

The checkpoint index reports 328,326,771,576 stored tensor bytes across 62 shards, 305.778 GiB total or an ideal four-way average of 76.445 GiB/rank. This is not a loaded-memory guarantee: replication, conversion, workspaces, KV, KDA state, and MTP scratch require separate allowance.

The pinned release has an official four-B200 TP4/EAGLE test. Independently, the passed GB300 FP8 canonical evaluation used TP4 with main/NextN weights 75.45/2.28 GiB per rank, 982 KDA slots consuming 34.78 GiB, 10.40 GiB MTP scratch at resident cap 48, 113.39 GiB target-plus-draft KV, 18,615,552 logical KV tokens, and 29.46 GiB free after graphs.

That GB300 allocation exposed 284,208 MiB/card, versus the B300 parent's 275,040 MiB/card. Its effective static fraction was 0.85595 versus this recipe's unchanged 0.85. Taking the approximately 9.26-GiB static-budget difference, another 10.19 GiB for cap 48→96 scratch, and an additional 10-GiB margin out of the observed GB300 KV budget leaves approximately 83.94 GiB, or 13.78M logical tokens at 6,540 bytes/token. This fixed-state projection is only a capacity estimate; graph sizes, kernel workspaces, and automatic state counts must be checked on B300.

A separate conservative bound starts with the B300 TP8 loaded weights doubled plus 10% allowance (86.4 GiB/rank), retains all 2,334 parent KDA slots at doubled per-state size (82.6 GiB), allows 20.55 GiB scratch at cap 96, and subtracts another 10 GiB from the 228.30-GiB static budget. Approximately 28.75 GiB remains for KV, about 4.72M logical tokens. The pinned joint KDA/state allocator can reduce persistent slot count; retaining every TP8 slot is deliberately conservative, not a prediction of the TP4 allocation.

Allocation and cost accounting

The node remains reserved with eight GPUs, exclusive allocation, and --mem=0; only four GPUs serve the model. Record configured_total=8, backend_total=4, actual AllocTRES, GPU visibility, host/cgroup headroom, and resolved native HiCache pool bytes rather than inferring them from the matrix.

dram-utilization: 1.0 preserves nominal budget bookkeeping. TP4's proportional legacy ON field is 1,499 GB; OFF's zero field is offload metadata, not zero host allocation. Neither limits actual RAM: the native 0.65 allocator sees the visible full-node host/cgroup budget and four local ranks.

Report both active-GPU and full-reserved-node economics. If aggregate throughput is R4, active-GPU normalization divides by four, while this eight-GPU reservation divides by eight. At the same per-GPU cost assumption the full-reserved-node cost-normalized throughput is half the active-GPU view; beating TP8 per active GPU requires R4 > R8/2, while beating TP8 for the entire reserved node requires R4 > R8. No second replica or automatic 2× cost improvement is assumed, and idle-GPU/node power must not be omitted from a full-node claim.

Validation gates

Local validation passes 121 focused tests and asserts exactly nine performance rows plus two canonical evaluation rows. All eleven bound recipes pass the real srtctl schema and production SGLang worker-command builder, with four active GPUs/eight reserved, common caps 96, ON nominal DRAM 1,499 GB, matched non-cache arguments, and exact measured AL 3.603536 only in performance mode. The production lm-eval function was locally captured with subprocess execution suppressed for OFF16 and ON96: both select the canonical GSM8K YAML and have no --limit. This is local command/schema evidence; the full sbatch CLI still requires compute-runtime binaries absent from the sandbox.

Native TP4 allocation checks and both canonical accuracy controls are now recorded above, while simulation-OFF forced host-restore proof remains outstanding. Keep the unresolved empty-content errors visible and phase-separated; neither shortening the full-context workload nor passing an OFF anchor resolves them. This draft contains no engine patch, precision change, or workload subset, and its completed GPU results remain qualified rather than an error-free production signoff.

中文

本草稿直接基于 #4731 的提交 4a3c3e8deb4bbca6c29b5dae16e2cb500bd0938e，不修改父分支。该配置同时调整 TP 与准入上限：所有点均为 TP4、最多 96 个驻留请求、decode CUDA graph 上限 96，因此不能将结果解释为仅改变 TP 的因果对照。

性能矩阵包含 OFF 并发 8 锚点，以及 OFF/ON 并发 48/64/80/96，共九点。另用独立 selector 保留全量标准 GSM8K OFF16、ON96 两个精度评测，均关闭合成接受率。保留原镜像、FP8、checkpoint 默认 MTP 精度、EAGLE 5/1/6、完整 393 条数据、3,600 秒正式运行和模型默认 1,048,576-token 上下文；ON 自动主机内存比例为 0.65，不设置固定 cache size/ratio。

checkpoint 总存储张量为 328,326,771,576 bytes，不能仅凭四路平均权重大小断言可运行。已有 GB300 TP4 标准评测实际加载主模型/NextN 为 75.45/2.28 GiB 每卡，resident cap 48 时有 18.62M 逻辑 KV tokens；本次 B300 实际日志进一步确认 924 个 KDA slots、15,815,936 逻辑 GPU KV tokens、同样的主模型/NextN 权重占用及 graph 后约 25 GiB 空闲显存。ON 原生自动预算约 625 GiB/rank，估计 host pools 为 299.3 GiB/rank；保持原生默认 ratio 2，并非 recipe 手动指定 ratio，也不等于名义 DRAM 字段（ON80 日志）。

实际预留仍为独占八 GPU 节点，仅四 GPU 参与推理。名义 ON DRAM 字段 1,499 GB 和 OFF 零值均不等于真实主机内存分配，需记录实际 AllocTRES、cgroup 与缓存池字节数。成本图必须同时说明四张活跃 GPU 和八张预留 GPU 两种口径：相同单卡价格下，整节点成本归一化吞吐是活跃 GPU 口径的一半，不假定第二副本或自动两倍成本收益。

本 PR 保持实验性草稿，用户授权的 full-sweep-fail-fast run 41352868379 已在 exact head 82c2ac3e3594cf8b12ca3a2399631dba34782d2a 完成全部九个性能点及两个标准评测。OFF8 canary 先于 gate 和其余任务完成，九个指纹均匹配；完整 393 条数据、3,600 秒命令和 1,048,576-token 上下文均保留。

性能结果中，48/64/80/96 并发的 OFF/ON 总输出 TPS 分别为 1,568.15/1,829.03、1,358.54/2,298.98、1,040.44/2,436.90、484.23/745.50。ON80 仅是输出 token 吞吐最高点；ON96 虽然完成但明显变慢，具体原因尚未证实。

总 token TCO 视图应使用输入加输出吞吐：ON48/64/80/96 分别为 271,093.61018 / 306,458.15223 / 287,825.17117 / 98,620.97659 TPS。总 token 吞吐最高的是 ON64，而非 ON80；ON64 的应用 P90 交互性也高于 ON80，43.898156 对 26.075619，因此在这两个轴上占优。应用 X 来自 1 / full_response_itl.p90，不直接使用导出的 interactivity 字段；ON64 在价格归一化之前，按四张活跃卡为 76,614.53806 TPS/卡，按八张预留卡为 38,307.26903 TPS/卡（总 token 结果）。

全量 GSM8K 评测均关闭合成接受率：OFF16 strict 为 1,284/1,319（97.3465%），flexible 为 1,283/1,319；ON96 strict 为 1,281/1,319（97.1190%），flexible 为 1,280/1,319。并发不同且差异样本数很小，不据此声称统计显著的精度变化。

原始 profiling 共 45,949 次请求，其中 45,942 成功、7 次空内容 InvalidInferenceResultError，另有 4,049 次 warmup 错误，必须分开报告；根因仍未解决。没有 context-overflow skip，最大观察输入为 847,410 tokens。ON 性能点均出现 host-cache hits，但这些性能运行使用合成接受率；未采集强制 SIM-OFF host restore 证明，普通标准评测通过不等于主机恢复路径已完全验证（运行产物）。

Profiling 阶段摘要还记录 JSONL 之外的 298 次请求取消：OFF8=0、OFF48=1、ON48=0、OFF64=12、ON64=5、OFF80=44、ON80=7、OFF96=109、ON96=120。这些值来自原始 Phase profiling ... complete 行的请求级 cancelled=，不混入另行记录的 session 计数；上述 45,949 条已完成记录不包含这些未完成请求（benchmark 日志）。

资源快照确认四 GPU 活跃、八 GPU 预留；九点 power_valid 均通过，但只验证四张活跃 GPU，不包含其他预留 GPU 或整机 CPU/DRAM 能耗。保留四卡活跃与八卡预留两种成本口径，不假设第二副本或自动两倍经济收益（运行产物）。

本地 121 项测试通过，计划严格生成九个性能点和两个标准评测点；十一个绑定配置均通过真实 schema 与生产 SGLang 命令构建器。OFF16 和 ON96 的标准 GSM8K 命令在禁止子进程执行时完成捕获，均无 --limit 且服务环境不含合成接受率。完整 sbatch CLI 仍缺少计算节点运行时二进制文件，以上本地验证不代表已启动 GPU 或完成实际精度评测。

AI model disclosure

GPT-6 Astra coordinated the experiment. A delegated GPT-6 Astra agent audited capacity and implemented this B300 FP8 TP4 child.

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41352868379

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41352868379

[Oseltamivir]

/use 41352868379

[github-actions]

@Oseltamivir staging run 41352868379. Existing staged runs will be preserved; a completion link will be posted here.

[Oseltamivir]

Superseded by #4710 (merged as 23a74c6), which consolidates GLM-5.3-Flash B300 FP8 + NVFP4 base and HiCache points into one PR (no TP4 on the 8-GPU node). / 已由 #4710（合并为 23a74c6）取代，该 PR 将 GLM-5.3-Flash B300 FP8 与 NVFP4 的基础点和 HiCache 点合并为一个 PR（8 卡节点不跑 TP4）。