[K3.1-Pro] vLLM B300: low-latency decode and SGLang draft depths / [K3.1-Pro] vLLM B300：启用低延迟 decode 并对齐 SGLang 草稿深度 by ZJY0516 · Pull Request #4833 · SemiAnalysisAI/InferenceX

ZJY0516 · 2026-11-05T16:06:29Z

Tunes kimik3.1-fp4-b300-vllm-agentic-dspark (Kimi-K3.1-Pro, B300, vLLM, DSpark):

Low-latency decode: --additional-config '{"decode_backend":"latency_specialized"}' at every point (the vLLM K3.1 recipe's fused CuTe DSL decode path).
c1-c2 at plain TP8 instead of TP8 x DCP8; c4 and up keep DCP8.
Draft depths update: DSpark 7 tokens through c8, 6 at c16, 5 at c24-c40, 4 from c48. Each decode graph cap covers max-num-seqs x (1 + drafts).
Startup timeout: VLLM_ENGINE_READY_TIMEOUT_S and the health check go to 7200 s, because the specialized path adds an autotune pass.

Local A/B on GB300 (TP8 over two trays, 20 min per point, previous draft depths):

conc	ITL p50 (main / this PR)	interactivity p50 (main / this PR)
1	5.32 / 2.77 ms	188 / 361 tok/s
32	17.86 / 15.16 ms	56 / 66 tok/s
70	37.09 / 33.09 ms	27 / 30 tok/s

All 11 points started and finished with the previous draft depths. The new draft depths are still being checked locally (c8, c56, c70).

中文

调整 kimik3.1-fp4-b300-vllm-agentic-dspark（Kimi-K3.1-Pro，B300，vLLM，DSpark）：

**低延迟 decode：**所有点启用 --additional-config '{"decode_backend":"latency_specialized"}'，即 vLLM K3.1 recipe 的融合 CuTe DSL decode 路径。
c1-c2 改用纯 TP8，不再用 TP8 x DCP8；c4 及以上保留 DCP8。
**草稿深度与 SGLang recipe 对齐：**DSpark 在 c8 及以下为 7 个 token，c16 为 6，c24-c40 为 5，c48 起为 4。每个 decode graph 上限覆盖 max-num-seqs x (1 + drafts)。
**启动超时：**specialized 路径多一轮 autotune，VLLM_ENGINE_READY_TIMEOUT_S 和 health check 提高到 7200 秒。

本地 GB300 A/B（两个 tray 组成 TP8，每点 20 分钟，旧草稿深度）见上表：c1 的 ITL p50 从 5.32 ms 降到 2.77 ms，c32 从 17.86 ms 降到 15.16 ms，c70 从 37.09 ms 降到 33.09 ms。

旧草稿深度下 11 个点都能正常启动并跑完；新草稿深度仍在本地验证（c8、c56、c70）。

🤖 Generated with Claude Code

https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

Turn on decode_backend=latency_specialized at every point, run c1-c2 at plain TP8 instead of TP8 x DCP8, and raise the c1 and c70 graph caps to 16 and 432 so the full decode step is captured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

…7200 s The latency_specialized decode path adds a second FlashInfer autotune pass at startup; on a local GB300 run it took ~40 min to become ready. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

DSpark now drafts 7 tokens through c8, 6 at c16, 5 at c24-c40 and 4 from c48, the same draft counts as the SGLang DFLASH recipe, and each decode graph cap covers max-num-seqs x (1 + drafts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

chatgpt-codex-connector · 2026-11-05T16:06:46Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-05T16:12:13.490626Z	`55582cd`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

chatgpt-codex-connector · 2026-11-05T16:12:12Z

💡 Codex Review

InferenceX/inferencex-e2e/perf-changelog.yaml

Line 12052 in 55582cd

pr-link: TBD

Use a supported changelog link placeholder

When maintainers run the required reuse merge, TBD is not one of PR_LINK_PLACEHOLDERS (XXX or the /pull/XXX URL): prepare_perf_changelog_merge.canonicalize_appended_links, which merge_with_reuse invokes, raises unexpected pr-link 'TBD'. This non-retirement benchmark change therefore cannot complete the mandated merge flow until this is replaced by a supported placeholder or the actual PR URL.

AGENTS.md reference: AGENTS.md:L89-L89

InferenceX/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/b300-fp4-mtp/agentic.yaml

Lines 62 to 66 in 55582cd

    
           # Admission is 2x CONC. DSpark drafts match the SGLang recipe's DFLASH drafts: 7 through c8, 
        
           # 6 at c16, 5 at c24-c40 and 4 from c48. The largest decode graph covers max-num-seqs x (1 + drafts), 
        
           # rounded up to a default capture size; throughput runs replace block rejection with the 
        
           # golden acceptance length. Plain TP8 at c1-c2, where the KV fits on GPU and DCP's decode 
        
           # collectives cost most; DCP8 from c4 for ~7x the KV tokens.

Remove the override-section header comment

These lines are a header for the complete override section and restate admission, draft counts, graph sizing, and the TP/DCP topology rather than briefly explaining one specific setting. The repository's mandatory comment convention explicitly forbids headers describing recipes, topologies, or overrides, so remove this summary and retain only value-specific rationale beside the affected keys.

AGENTS.md reference: AGENTS.md:L54-L58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ency-decode # Conflicts: # inferencex-e2e/perf-changelog.yaml

github-actions · 2026-11-05T23:26:36Z

Dispatched trusted sweep run for approved external revision 5c315233e4c09e69ec1607a4f90170272b5e523c.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 5c315233e4c09e69ec1607a4f90170272b5e523c 调度trusted sweep run。
后续新提交不会自动获得信任；如需批准新的 SHA，请移除并重新添加主扫描标签。

Resolve the perf-changelog conflict by keeping main's new entries and appending the B300 vLLM entry last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

The kimi-k31 image launches the wide (65-320 row) MoE experts on 152 CTAs that must all be co-resident; a B300 has 148 SMs, so every point whose decode graphs exceed 64 rows (c8-c70) failed engine init. Keep latency_specialized where graphs stay within 64 rows and run the stock decode above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

github-actions · 2026-11-06T02:25:40Z

Dispatched trusted sweep run for approved external revision 947a5e905f231580d40cd602dae7c8645745ee0e.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 947a5e905f231580d40cd602dae7c8645745ee0e 调度trusted sweep run。
后续新提交不会自动获得信任；如需批准新的 SHA，请移除并重新添加主扫描标签。

cquil11 · 2026-11-06T06:46:57Z

/use 42150239586

github-actions · 2026-11-06T06:47:14Z

@cquil11 staged run 42150239586.

Kimi-K3.1-Pro · agentic-traces · FP4 · b300_vllm · 2026-11-06

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

cquil11 · 2026-11-06T08:43:38Z

/use 42150239586

github-actions · 2026-11-06T08:43:52Z

@cquil11 staged run 42150239586.

Kimi-K3.1-Pro · agentic-traces · FP4 · b300_vllm · 2026-11-06

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…ency-decode # Conflicts: # inferencex-e2e/perf-changelog.yaml

github-actions · 2026-11-06T09:01:22Z

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped
benchmark	11	0	12
eval	1	1	1
trace	11	0	0
power	11	0	0

_{benchmarkFiles 12, rawBenchmarkRows 23, coveredByPerJobFiles 12, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 1319, gpuMetricSamplesNew 554880, powerPublicationPoints 11, changelogEntriesWritten 1}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

ZJY0516 and others added 3 commits November 5, 2026 10:04

ZJY0516 requested a review from a team November 5, 2026 16:06

ZJY0516 requested review from Ankur-singh, RohitNagraj, kedarpotdar-nv, nvpohanh and wzhao18 as code owners November 5, 2026 16:06

github-project-automation Bot added this to InferenceX Board Nov 5, 2026

claude Bot reviewed Nov 5, 2026

View reviewed changes

chore(kimik3.1): link the B300 vLLM changelog entry to its PR

05a295f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

functionstackx mentioned this pull request Nov 5, 2026

[Klaud Cold][K3.1-Pro] vLLM B300: low-latency decode and SGLang draft depths / [Klaud Cold][K3.1-Pro] vLLM B300：启用低延迟 decode 并对齐 SGLang 草稿深度 #4838

Closed

cquil11 mentioned this pull request Nov 5, 2026

[K3.1-Pro] Re-run vLLM on B300 with synthetic parsers off #4841

Closed

Merge remote-tracking branch 'origin/main' into kimik31-vllm-b300-lat…

168a0aa

…ency-decode # Conflicts: # inferencex-e2e/perf-changelog.yaml

cquil11 added the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Nov 5, 2026

Merge remote-tracking branch 'origin/main' into kimik31-vllm-b300-lat…

5c31523

…ency-decode # Conflicts: # inferencex-e2e/perf-changelog.yaml

cquil11 added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Nov 5, 2026

ZJY0516 and others added 2 commits November 6, 2026 02:16

Merge main into kimik31-vllm-b300-latency-decode

6b4c106

Resolve the perf-changelog conflict by keeping main's new entries and appending the B300 vLLM entry last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188HSzNcNLyDbocjggjUSvq

ZJY0516 added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Nov 6, 2026

Merge remote-tracking branch 'origin/main' into kimik31-vllm-b300-lat…

e26a37f

…ency-decode # Conflicts: # inferencex-e2e/perf-changelog.yaml

cquil11 merged commit af33fcb into SemiAnalysisAI:main Nov 6, 2026
29 checks passed

github-project-automation Bot moved this to Done in InferenceX Board Nov 6, 2026

claude Bot mentioned this pull request Nov 6, 2026

perf(amd): Kimi-K3.1 MI355X vLLM agentic: 898 GB CPU offload, mbt 8192, capture cap 8192 #4867

Closed

abmfy mentioned this pull request Nov 7, 2026

[K3.1-Pro] vLLM B300: per-concurrency TP8 / DCP8 / TEP recipe on kimi-k31-mc + TEP shared-expert TP shard / [K3.1-Pro] vLLM B300：按并发选择 TP8 / DCP8 / TEP，基于 kimi-k31-mc 并对 TEP shared expert 做 TP 切分 #4878

Draft

Navigation Menu

Repository navigation

[K3.1-Pro] vLLM B300: low-latency decode and SGLang draft depths / [K3.1-Pro] vLLM B300：启用低延迟 decode 并对齐 SGLang 草稿深度 - #4833

[K3.1-Pro] vLLM B300: low-latency decode and SGLang draft depths / [K3.1-Pro] vLLM B300：启用低延迟 decode 并对齐 SGLang 草稿深度#4833
cquil11 merged 9 commits into
SemiAnalysisAI:mainfrom
ZJY0516:kimik31-vllm-b300-latency-decode

ZJY0516 commented Nov 5, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026

Uh oh!

github-actions Bot commented Nov 5, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

cquil11 commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

cquil11 commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ZJY0516 commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Nov 5, 2026

💡 Codex Review

Uh oh!

github-actions Bot commented Nov 5, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

cquil11 commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cquil11 commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZJY0516 commented Nov 5, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 5, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading