Repository navigation
[K3.1-Pro] Day-0 support SGLang on GB200 - #4804
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - kimik3.1-fp4-gb200-dynamo-sglang-agentic-dflash-disagg | ||
| description: | ||
| - "Add Kimi K3.1 Pro day-0 SGLang AgentX points on GB200: aggregated TP16 DCP16 and 1P1D TP16 DCP16, with RadixArk DFLASH block verification and HiCache CPU offload." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The changelog CI check will reject this PR because the new block's pr-link is TBD. The validator only accepts the canonical PR URL or an XXX placeholder. validate_added_pr_link (infx/workflows/validate_perf_changelog.py:113) raises "new PR entry must use ... or an XXX placeholder". The merge preparer also refuses TBD at prepare_perf_changelog_merge.py:108. Fix: set pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or use the real PR number, so the merge step can rewrite it.
Why this was flagged
The new block at perf-changelog.yaml:12021 has pr-link: TBD. On a PR run, validate_perf_changelog.py:113 checks every appended entry: link not in PR_LINK_PLACEHOLDERS and link != expected raises ChangelogValidationError. PR_LINK_PLACEHOLDERS (line 21) contains only "XXX" and "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX". On main, CANONICAL_PR_LINK.fullmatch fails for TBD. prepare_perf_changelog_merge.py:108-111 raises "appended entry N has unexpected pr-link 'TBD'", so the automatic link rewrite also fails. No other changelog entry in the file uses TBD, so this PR's changelog validation fails and merge preparation is blocked.
Verification: The diff appends a block that ends with pr-link: TBD (perf-changelog.yaml, new tail block). PR_LINK_PLACEHOLDERS (line 21) holds only "XXX" and "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX". As a result, the check-changelog gate fails and the sweep for the new config keys does not run until the link is changed to XXX or to the real PR URL.
| dyn-reasoning-parser: kimi_k3 | ||
| dyn-tool-call-parser: kimi_k3 | ||
| speculative-algorithm: DFLASH | ||
| speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview |
There was a problem hiding this comment.
🔴 Kimi K3.1 GB200 worker startups can fail with Hub 429 errors, because every rank fetches the DFLASH draft from Hugging Face at launch. The roles name RadixArk/Kimi-K3.1-Dflash2-Preview but set neither HF_HUB_OFFLINE nor HF_HUB_CACHE. Muse Spark DFlash2 recipes added both after concurrent launches hit the Hub 429 limit. Fix: give every speculating role HF_HUB_OFFLINE: '1' and HF_HUB_CACHE: /hf_hub_cache, and seed the draft in gb200-nv's shared hf-hub-cache; this covers the 3 sites listed. Same pattern at 3 sites (agg-variants.yaml:68, disagg-variants.yaml:75, disagg-variants.yaml:128).
Why this was flagged
The trigger is a sweep that launches several of the 10 new points at once (7 agg and 3 disagg), or gb200-nv compute nodes with no Hub access. With no HF_HUB_OFFLINE, every rank resolves speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview (agg-variants.yaml:68, disagg-variants.yaml:75 and :128) through Hub API calls. The role env blocks set no HF_HUB_CACHE. As a result the draft downloads into each container's default cache, not the shared /hf_hub_cache mount that gb200-nv's agentic-volume-mounts provides (runners.yaml:623-625). The launcher passes HF_HUB_CACHE only to the benchmark client (config.py:236-252). perf-changelog.yaml:10160-10205 records that the Muse Spark DFlash2 points needed HF_HUB_OFFLINE=1 because concurrent launches otherwise hit the Hub 429 API rate limit. Neither safeguard applies to these new recipes, so a rate-limited or offline rank fails worker startup and the point produces no result.
Verification: Every speculating role names a Hub repo id as its draft: speculative-draft-model-path: RadixArk/Kimi-K3.1-Dflash2-Preview (agg-variants.yaml:68, disagg-variants.yaml:75 and :128). None of the role env blocks sets HF_HUB_OFFLINE or HF_HUB_CACHE, and neither file has a base environment: block. The srt driver does not add them for workers either.
3b0bf0e to
5ff93c4
Compare
Aggregated TP16 DCP16 and 1P1D TP16 DCP16 dynamo-sglang points on gb200-nv with RadixArk DFLASH block verification, HiCache CPU offload and Mooncake PD transfer, at Kimi K3's GB200 concurrencies.
Remove the DCP comm backend and the custom all-reduce and DCP push-combine env vars, which the SGLang cookbook's GB200 command does not use.
5ff93c4 to
3f7acfd
Compare
cquil11
added
the
non-canary-full-sweep-enabled
The SGLang image ships no Dynamo, and published Dynamo wheels cannot load the kimi_k31 tokenizer or accept DCP decode for K3.1's DSA layers. Install ai-dynamo/dynamo#18124, which fixes both, and record it as the router version.
…lang # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41990398212 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41990398212 |
No Dynamo revision both builds and loads the kimi_k31 tokenizer, so the workers run sglang.launch_server behind the image's sglang-router 0.3.2. The configs become plain sglang (keys drop dynamo-), the Dynamo services and router args go, the reasoning and tool-call parsers move to SGLang's own flags, and the gb200-nv AgentX power rule accepts sglang. Engine args, topology, offload and forced acceptance are unchanged.
At TP16 the K3 sharded MoE front builds an N=1048 slice GEMM that the warp split-K kernel has no output tile for, so every worker capturing graphs above 17 tokens crashed. SGLANG_K3_SHARDED_MOE=0 selects the engine's replicated fused front; the front stays BF16 and K3 all-reduce fusion stays on.
The K3.1 SGLang image ships Dynamo 1.6.0, whose SGLang argument check lets Kimi-K3.1 decode with DCP on its DSA backend. Following Po-Han Huang's GB300 recipes, the frontend runs the KV router with SGLang's chat processor, which renders kimi_k31 requests the frontend tokenizer cannot load; the aggregated and prefill workers publish KV events and advertise session affinity. The configs return to dynamo-sglang with the image's dynamo-router 1.6.0. Engine args, topology, offload and forced acceptance are unchanged.
The workers had no HF_HUB_CACHE, so every node downloaded the DFlash2 draft into its container on each start. Load times varied from seconds to minutes, and agg c1 and c24 failed when one node finished more than SGLang's 480 s post-load barrier after the others. Point the workers at the mounted shared cache, as Po-Han's GB300 recipes and other GB300 SGLang recipes on main do.
…lang # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/use 41990398212 |
|
@cquil11 staged run 41990398212. This run remains available across future |
|
/reuse-sweep-run 41990398212 |
|
/reuse-sweep-run 41990398212 |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 10, rawBenchmarkRows 10, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 9233, gpuMetricSamplesNew 1142864, powerPublicationPoints 10, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds Kimi K3.1 Pro day-0 SGLang AgentX points on GB200 NVL72 (
cluster:gb200-nv,dynamo-sglang). They cover the same concurrencies and CPU KV offloading that Kimi K3 has on InferenceX for GB200. Speculation uses the RadixArk DFLASH draft with block verification.kimik3.1-fp4-gb200-dynamo-sglang-agentic-dflash-aggkimik3.1-fp4-gb200-dynamo-sglang-agentic-dflash-disagglmsysorg/sglang:dev-kimi-k31-pro@sha256:67fc77d1…(multi-arch).dynamo.install: false) with the KV router, session affinity and SGLang's chat processor (dyn-chat-processor: sglang), following Po-Han Huang's GB300 Dynamo SGLang recipes ([NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: DFLASH disaggregated and aggregated megakernel points / [NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang：DFLASH 分离式与聚合式 megakernel 配置点 #4811). The image's Dynamo accepts K3.1 DSA decode at DCP16.SGLANG_K3_SHARDED_MOE=0. SGLang's K3 sharded MoE front has no warp split-K output tile for TP16's N=1048 (k3_ar_fusion.slice_gemm), and workers crashed during CUDA-graph capture. The engine's replicated fused front stays a BF16 GEMM, and K3 all-reduce fusion stays on.RadixArk/Kimi-K3.1-Dflash2-Previewwith--speculative-use-block-verification, FP8 draft KV andcutedsl_mladraft attention.--enable-linear-replayssm-specis set on the verifying roles only, and both PD roles use the same verification width, as the cookbook's PD cells do.SGLANG_SIMULATE_ACC_LEN) on each speculating role. Nothing is hard-coded.max-running-requestsis 2× concurrency.dram-utilization: 0.54gives each tray a 487 GB cap, which covers 4 ranks × 120 GB of HiCache.kimik3.1joins the gb200-nv shared run root, as Kimi K3 did, and the existing GLM-5.2 FP4dynamo-sglangAgentX power rule.AIPERF_EXTRA_INPUTS, matching the K3.1 vLLM recipes, so both engines get identical client requests.K3.1 week-1 precision policy:
kv-cache-dtype: fp8_e4m3.mamba-ssm-dtype: float32. Conv cache stays at the native BF16 default.--enable-symm-memare not used.Risks:
install-torchao.shsetup step (an unpinnedpip install torchao) also runs here.