Repository navigation
[K3.1-Pro] Day-0 support SGLang on B200 - #4801
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| description: | ||
| - "Add Kimi K3.1 Pro B200 SGLang AgentX on lmsysorg/sglang:dev-kimi-k31-pro: one aggregated worker over two nodes, TP16 with DCP16, no KV offload, concurrency 1, 4, 8, 14, 24, 32, 48, 56, 72, 80 with DFLASH and 96 without speculation." | ||
| - "DFLASH uses RadixArk/Kimi-K3.1-Dflash2-Preview with block verification, 7 draft tokens through concurrency 14 and 4 above." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The perf-changelog check will fail on this PR, so the PR stays red until the link is fixed. The new entry uses pr-link: TBD. The validator accepts only the PR's own canonical URL or an XXX placeholder (PR_LINK_PLACEHOLDERS in validate_perf_changelog.py:21). Fix: set pr-link to https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or to this PR's real URL.
Why this was flagged
On a PR run, validate_added_pr_link (infx/workflows/validate_perf_changelog.py:105-116) compares each appended entry's link against expected (the PR's own URL) and PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}. "TBD" is in neither, so it raises new PR entry must use '...' or an XXX placeholder; found 'TBD'. That fails the changelog validation job for this PR. On a main-branch run, CANONICAL_PR_LINK would also reject it with new main-branch entry has invalid pr-link. No other entry in perf-changelog.yaml uses TBD.
Verification: At perf-changelog.yaml:12023 the PR appends pr-link: TBD. validate_added_pr_link (validate_perf_changelog.py:105-116) raises because "TBD" is not in PR_LINK_PLACEHOLDERS. The PR-run check (run-sweep.yml:96-116) never calls it, so the failure lands at merge instead. merge_with_reuse.py:688 reaches compare_entries (prepare_perf_changelog_merge.py:88), which raises on 'TBD'.
| precision: fp4 | ||
| framework: dynamo-sglang | ||
| multinode: true | ||
| disagg: false | ||
| srt-recipe-dir: kimik3.1/sglang/b200-fp4/agentx | ||
| scenarios: | ||
| agentic-coding: | ||
| - search-space: | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [1] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c1 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| dcp-size: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [4] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c4 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| dcp-size: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [8] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c8 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| dcp-size: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [14] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c14 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| dcp-size: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [24] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c24 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| dcp-size: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [32] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c32 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| dcp-size: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| - spec-decoding: draft_model | ||
| kv-offloading: none | ||
| conc-list: [48] | ||
| num-nodes: 2 | ||
| srt-recipe: agg-variants.yaml:override_tp16_c48 | ||
| power: true | ||
| worker: | ||
| num-worker: 1 |
There was a problem hiding this comment.
🔴 If this merges before #4798, all ten DFLASH rows fail at binding with No golden curve for model prefix 'kimik3.1' and method 'dflash'. Only the c96 no-spec row would launch. identities.yaml has no kimik3.1 dflash identity, and this diff does not add one. Fix: add the kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block identity and its curve in this change. Otherwise, gate the merge on #4798 or leave the DFLASH rows out until it lands.
Why this was flagged
The condition is merging this PR before the stacked #4798. Every search-space row with spec-decoding: draft_model (nvidia-master.yaml:4817-4944) resolves a golden AL curve by model prefix kimik3.1 and method dflash. infx/golden_al_distribution/identities.yaml has kimik3.1 only under method dspark (line 105-106). The only dflash identity there is for musespark (line 170-171). identity.py:260 therefore raises ValueError("No golden curve for model prefix 'kimik3.1' and method 'dflash'") for 10 of the 11 rows, so the sweep produces only the c96 no-spec point. The PR description acknowledges the dependency, but nothing in this diff enforces the merge order.
Verification: The diff does not touch infx/golden_al_distribution/identities.yaml. agg-variants.yaml sets speculative-algorithm: DFLASH (line 50). identities.yaml selects kimik3.1 only with method: dspark (line 106), so identity.py:260 raises ValueError. Result: all ten spec-decoding: draft_model rows fail to launch.
968b4d9 to
09310d5
Compare
One aggregated SGLang worker over two B200 nodes (TP16 + DCP16) serving the public port directly, with RadixArk DFLASH and block verification at concurrency 1-80 and no speculation at 96.
09310d5 to
e80fb62
Compare
cquil11
added
the
non-canary-full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41976897857 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41976897857 |
…ng AgentX client The aggregated SGLang worker serves the public port without a Dynamo frontend, and agentx profile rejects --use-dynamo-conv-aware-routing, so every throughput point failed before sending a request.
|
/use 41976897857 |
|
@cquil11 staged run 41976897857. This run remains available across future |
|
/reuse-sweep-run 41976897857 |
|
/reuse-sweep-run 41976897857 |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 11, rawBenchmarkRows 11, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 9233, gpuMetricSamplesNew 1261232, powerPublicationPoints 11, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds Kimi K3.1 Pro AgentX on B200 with SGLang, at the B200 points Kimi K3 shows on InferenceX today. One 8xB200 node cannot hold the ~1.56 TB MXFP4 checkpoint, so every point is one aggregated worker across two nodes.
The DFLASH points need #4798's golden curve before they can launch. Until
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-blocklands, binding these rows fails withNo golden curve for model prefix 'kimik3.1' and method 'dflash'. The curve is being collected in run 41910694249.The layout is unverified. The SGLang Kimi K3.1 Pro cookbook's B200 command (
--tp-size 16 --dcp-size 16) is marked as a placeholder: no K3.1 recipe has run on B200 yet.kimik3.1-fp4-b200-dynamo-sglang-agentic-dflashlmsysorg/sglang:dev-kimi-k31-pro@sha256:67fc77d1…, the SGLang team's K3.1 day-0 build.frontend.type: sglang, so the single aggregated worker serves the public port itself, the way Kimi K3's B200 points served vLLM directly. The cookbook's Dynamo snapshot needed a patched Dynamo for K3.1, so this avoids Dynamo. Thedynamo-sglangframework label only routes the point through theb200-nscalemulti-node lane. With no Dynamo frontend, the AgentX client runs withAIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0'.RadixArk/Kimi-K3.1-Dflash2-Previewwith--speculative-use-block-verification, FP8 draft KV,cutedsl_mladraft attention and--enable-linear-replayssm-spec.--speculative-num-draft-tokensis the verification width, so it is 8 for 7 drafts and 5 for 4 drafts; 5 is the draft's own configured block. The driver injects the golden AL; nothing is hard-coded.max-running-requestsis twice the concurrency, with decode CUDA graphs to the same batch.SGLang caps max_running_requests at 40 on the DFLASH points and 60 at c96 (mamba state cache, 5 slots per request), so points above c40 measure saturation.
power.pygains the matchingb200-nscalemulti-node rule.K3.1 week-1 precision policy:
kv-cache-dtype: fp8_e4m3.mamba-ssm-dtype: float32.Stacked on #4798 (DFLASH golden curve), so the diff includes its commits until it merges.