Repository navigation
[Klaud Cold][K3.1-Pro] vLLM B300: low-latency decode and SGLang draft depths / [Klaud Cold][K3.1-Pro] vLLM B300：启用低延迟 decode 并对齐 SGLang 草稿深度 - #4838
Conversation
… depths / [Klaud Cold][K3.1-Pro] vLLM B300：启用低延迟 decode 并对齐 SGLang 草稿深度 Duplicate of #4833 (kimik31-vllm-b300-latency-decode) rebuilt on current main. The perf-changelog entry is appended after main's newest entries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Besides the inline finding, I checked two other things and found no problem with either:
- Splitting the matrix row:
infx/srt_slurm/single_node.pypicks the recipe variant by CONC and bydecode-context-parallel-sizeagainst DCP_SIZE, not by override name. So the conc 1-2 row with no dcp-size gets exactly one match (override_tp8_c1/c2), and the conc 4-70 dcp-size 8 rows each get exactly oneoverride_dcp8_*. - Decode graph caps: each new
max-cudagraph-capture-sizeis at leastmax-num-seqs x (1 + num_speculative_tokens). For example, c70 needs 140 x 5 = 700, and the cap is 704.
Extended reasoning...
This is a config-only change to one vLLM AgentX recipe, its nvidia-master search-space row and a perf-changelog entry. It touches no security-sensitive code. The checks above cover how a recipe variant is chosen and how the CUDA-graph caps are sized. The pr-link problem is covered separately in the inline comment.
| description: | ||
| - "Kimi-K3.1-Pro B300 vLLM: enable the recipe's low-latency decode (decode_backend=latency_specialized) at every point and run c1-c2 at plain TP8 instead of TP8 x DCP8." | ||
| - "Match the SGLang recipe's draft depths (DSpark 7 tokens through c8, 6 at c16, 5 at c24-c40, 4 from c48), size each decode graph cap to max-num-seqs x (1 + drafts), and raise the engine-ready timeout and health check to 7200 s." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The changelog check will fail on this PR, and the merge-prep step will refuse it. pr-link: TBD is not a value either tool accepts. validate_added_pr_link only takes the PR's own URL or a value in PR_LINK_PLACEHOLDERS ("XXX" or ".../pull/XXX"). Fix: set the appended entry's pr-link to this PR's canonical URL, or to https://github.com/SemiAnalysisAI/InferenceX/pull/XXX so the merge step can fill it in.
Why this was flagged
The PR appends a new entry ending in pr-link: TBD at perf-changelog.yaml:12087. On a PR run, inferencex-e2e/infx/workflows/validate_perf_changelog.py:113 rejects any link that is neither in PR_LINK_PLACEHOLDERS (lines 21-24: "XXX" and "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX") nor the expected PR URL. It raises "new PR entry must use ... or an XXX placeholder; found 'TBD'". prepare_perf_changelog_merge.py:108 also raises "appended entry ... has unexpected pr-link 'TBD'", so the reuse-assisted merge cannot canonicalize the link. Before this PR, the base changelog passes both checks. The TBD value matches the guidance in .claude/commands/add-model-hardware.md:144, but neither validator accepts it.
Verification: The diff appends a new block that ends in pr-link: TBD (perf-changelog.yaml, new last line). compare_entries in inferencex-e2e/infx/workflows/validate_perf_changelog.py:158-159 calls validate_added_pr_link on every appended entry. The changelog check fails and the merge-prep step will not proceed until pr-link is set to this PR's URL or an XXX placeholder.
…1-vllm-b300-latency-decode # Conflicts: # inferencex-e2e/perf-changelog.yaml
functionstackx
added
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42078563749 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42078563749 |
Duplicate of #4833 (
ZJY0516, branchkimik31-vllm-b300-latency-decode) rebuilt on current main so a same-repository sweep runs; #4833 is conflicting against main. Recipe, master-config and changelog content are unchanged apart from the changelog entry's position and pr-link.Tunes
kimik3.1-fp4-b300-vllm-agentic-dspark(Kimi-K3.1-Pro, B300, vLLM, DSpark):--additional-config '{"decode_backend":"latency_specialized"}'at every point (the vLLM K3.1 recipe's fused CuTe DSL decode path).max-num-seqs x (1 + drafts).VLLM_ENGINE_READY_TIMEOUT_Sand the health check go to 7200 s, because the specialized path adds an autotune pass.Local A/B on GB300 (TP8 over two trays, 20 min per point, previous draft depths):
All 11 points started and finished with the previous draft depths. The new draft depths are still being checked locally (c8, c56, c70).
中文
调整
kimik3.1-fp4-b300-vllm-agentic-dspark（Kimi-K3.1-Pro，B300，vLLM，DSpark）：--additional-config '{"decode_backend":"latency_specialized"}'，即 vLLM K3.1 recipe 的融合 CuTe DSL decode 路径。max-num-seqs x (1 + drafts)。VLLM_ENGINE_READY_TIMEOUT_S和 health check 提高到 7200 秒。本地 GB300 A/B（两个 tray 组成 TP8，每点 20 分钟，旧草稿深度）见上表：c1 的 ITL p50 从 5.32 ms 降到 2.77 ms，c32 从 17.86 ms 降到 15.16 ms，c70 从 37.09 ms 降到 33.09 ms。
旧草稿深度下 11 个点都能正常启动并跑完；新草稿深度仍在本地验证（c8、c56、c70）。
AI model disclosure
claude-fable-5-1), via Claude Code.🤖 Generated with Claude Code