Repository navigation
[K3.1-Pro] Re-sweep vLLM on GB200 with the updated DSpark draft - #4864
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Besides the inline finding, I checked whether the disaggregated row also needs the keep-alive setting. It doesn't: disagg-variants.yaml:48 already sets VLLM_HTTP_TIMEOUT_KEEP_ALIVE: "600", so this change brings the aggregated recipe in line with it.
Extended reasoning...
The keep-alive env var is the only change to the disaggregated side that I examined and ruled out. The disaggregated recipe already had the same value before this PR.
| - config-keys: | ||
| - kimik3.1-fp4-gb200-vllm-agentic-dspark-tp8pp2-agg | ||
| - kimik3.1-fp4-gb200-vllm-agentic-dspark-tp16-disagg | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Re-sweep Kimi-K3.1-Pro GB200 vLLM AgentX with the updated DSpark draft (Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a) and the #4851 golden curve, so throughput forces AL 3.41 at 5 draft tokens and 3.19 at 4." |
There was a problem hiding this comment.
🟡 (optional) The changelog says this sweep uses draft Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a, but no recipe pins that revision. Both rows load whatever the Hub default branch serves. The vLLM speculative-config JSON in agg-tp8pp2-variants.yaml:89-137 and disagg-variants.yaml:79,107 has no revision key. So the recorded points are tied to an unpinned draft, and nothing in either config key actually changes for this re-sweep. Fix: add "revision":"<full 6a2ef20a sha>" to every DSpark speculative-config in both GB200 recipes. This matches the speculative-draft-model-revision pin in the sglang K3.1 recipes.
Why this was flagged
Both listed keys start vLLM with speculative-config {"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview",...} and no revision (agg-tp8pp2-variants.yaml:89-137, disagg-variants.yaml:79,107). The draft that actually loads is whatever the nodes' HF cache or the Hub default branch gives at run time. Per the PR description, that already changed once without any config change. If Inferact pushes again, or a node keeps a stale cache, the results under this changelog entry and the 3.41/3.19 golden-curve forcing describe a different draft than the one that ran. The mismatch is silent. The base branch has the same unpinned load, but this entry is the first to claim a specific revision 6a2ef20a. The sglang K3.1 recipes do pin their draft with speculative-draft-model-revision (e.g. sglang/gb300-fp4/agentx/disagg-tp16-variants.yaml:110), so the vLLM ones are the odd ones out.
Verification: The new changelog block (perf-changelog.yaml:12295) says the re-sweep uses "Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a", but neither recipe pins that revision. Every DSpark speculative-config in agg-tp8pp2-variants.yaml and disagg-variants.yaml has no revision key. The unpinned config is pre-existing in the recipes.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42203512960 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42203512960 |
|
/use 42203512960 |
|
@cquil11 staged run 42203512960. This run remains available across future |
|
/reuse-sweep-run 42203512960 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 10, rawBenchmarkRows 10, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 9233, gpuMetricSamplesNew 1532268, powerPublicationPoints 10, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Re-sweeps Kimi-K3.1-Pro GB200 vLLM AgentX, both the aggregated and disaggregated rows from #4799, with the updated DSpark draft.
Inferact/Kimi-K3.1-Pro-DSpark-preview@6a2ef20a), and chore(golden-al): update Kimi K3.1 Pro DSpark probabilistic-block curve for draft 6a2ef20a #4851 updated the golden acceptance curve to match. [K3.1-Pro] Day-0 support vLLM on GB200 #4799 was swept before chore(golden-al): update Kimi K3.1 Pro DSpark probabilistic-block curve for draft 6a2ef20a #4851, so its throughput points were forced to the old values: 3.08 at 5 draft tokens and 2.92 at 4. This sweep uses the new values, 3.41 and 3.19. The GB200 nodes already load the new draft, which is why [K3.1-Pro] Day-0 support vLLM on GB200 #4799's eval AL gates read about 9% above the old curve.VLLM_HTTP_TIMEOUT_KEEP_ALIVE: "600"on the aggregated server, matching the disaggregated recipe. AIPerf reuses a session's connection across turns, and uvicorn's 5 s idle timeout caused oneClientOSErrorper point at c24–c56 in [K3.1-Pro] Day-0 support vLLM on GB200 #4799.