Repository navigation
[NVIDIA][AgentX] Muse Spark Open GB300 SGLang megakernel point at concurrency 1 / [NVIDIA][AgentX] Muse Spark Open GB300 SGLang megakernel 并发 1 测试点 - #5063
Conversation
…cy 1 Add musespark-fp4-gb300-dynamo-sglang-agentic-dflash-mk-agg, one TP8/EP8 Dynamo+SGLang worker over two GB300 nodes running the sgl-muse-mega NVIDIA megakernel on lmsysorg/sglang:dev-0775fc6-20261112-cu13-arm64. The point uses a linear DFlash2 chain of 7 drafts on the committed golden curve, greedy requests, a BF16 KV cache and a 1M context length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3965 | ||
|
|
||
| - config-keys: | ||
| - musespark-fp4-gb300-dynamo-sglang-agentic-dflash-mk-agg |
There was a problem hiding this comment.
🔴 Dashboard readers lose the published Muse Spark GB300 Dynamo-SGLang AgentX curve after merge; it shows just one greedy megakernel c1 point. The new key shares model-prefix, hardware, framework and precision with musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg. The repo says the newest normal run replaces that whole curve, and this regular entry sweeps only conc 1. Fix: give this point a curve identity separate from the existing family. If that is out of scope, list the sibling keys in this entry so the sweep reruns the whole curve. The PR text says the run does not extend that curve, but it replaces it.
Why this was flagged
infx/klaud/main.py:173 says a curve is every master key of one file sharing model, hardware, framework and precision, "since the newest run replaces that whole curve". So the new key at nvidia-master.yaml:7053 lands in the same agentic, offload-off agg series as dflash-agg override_tp8_c36 and override_tp8_c1 (nvidia-master.yaml:6992-7011). The changelog entry at perf-changelog.yaml:12847-12855 is a regular entry (no append-only) that lists only the new key. Its sweep yields one c1 point with greedy requests and a bf16 KV cache. On ingest that run becomes the newest run for the curve, so the c36 and standard sampled c1 points stop showing. On the base branch they stay published. The repo has handled this before: perf-changelog.yaml:12212 and 12828 list sibling agg keys "so the whole Dynamo + SGLang curve ... comes from this sweep".
Verification: Trigger: the regular changelog entry is merged and ingested. infx/klaud/main.py:171-173 and .github/klaud-candidate-prompt.md:69 both say "the newest normal run replaces that whole curve". This PR's regular entry (perf-changelog.yaml:12847-12855) sweeps only conc-list: [1]. That replaces the default-displayed multi-point curve.
There was a problem hiding this comment.
Confirmed and fixed in a7374bd. live_families puts the new -mk-agg key in one curve with both musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg and -disagg, so the c1-only run would have replaced the agg and the disagg points. The changelog entry now lists all three keys, and the sweep reruns the full Muse Spark GB300 Dynamo + SGLang curve (15 points), as #4673 and #4997 did. The sibling keys' image, recipes and points are unchanged, and the PR description no longer claims the run leaves that curve alone.
中文
已确认并在 a7374bd 中修复。live_families 会把新的 -mk-agg key 与 musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg 和 -disagg 归为同一条曲线,因此只跑 c1 的运行会替换掉 agg 和 disagg 的所有点。现在 changelog 条目列出全部三个 key,sweep 会像 #4673 和 #4997 一样重跑整条 Muse Spark GB300 Dynamo + SGLang 曲线(15 个点)。两个已有 key 的 image、recipe 和点都不变,PR 描述中也已删除"不影响该曲线"的说法。
| # Muse Spark Open on the sgl-muse-mega NVIDIA megakernel (--enable-nvidia-megakernel): one TP8 | ||
| # worker over two GB300 nodes. The plugin fuses each batch-1 decode step (DFlash2 draft, target | ||
| # verify and accept) into one CUDA graph; prefill and larger batches run SGLang's own path. |
There was a problem hiding this comment.
🟡 (optional) Maintainers get two new header comments, which the repo comment rule bans; one also carries build history. The recipe header at agg-mk-variants.yaml:3-5 describes the recipe, topology and plugin. The master-config header at nvidia-master.yaml:7048-7052 also names source commits and where ai-dynamo came from. Fix: delete both header blocks, which covers the 2 sites listed. Keep the line-level comments that explain specific values, and leave the image and commit details in perf-changelog.yaml and the PR description. Same pattern at 2 sites (agg-mk-variants.yaml:3, nvidia-master.yaml:7048). [also at: inferencex-e2e/configs/nvidia-master.yaml:7052 - nit: Maintainers get two header comments that the AGENTS.md comment rule forbids, and one of them records build provenance.]
Why this was flagged
AGENTS.md:56-57 bans header comments that describe a file, topology, config, recipe, override or section, and bans history such as where code came from. agg-mk-variants.yaml:3-5 is a block above base: that describes the plugin and the 'one TP8 worker over two GB300 nodes' topology, so it is a recipe and topology header. nvidia-master.yaml:7048-7052 sits above the musespark-fp4-gb300-dynamo-sglang-agentic-dflash-mk-agg key. It describes the config and records provenance: 'SGLang 5884e4fec with the plugin at 0775fc6' and 'ai-dynamo 1.7.0.dev20261026 from the base nightly'. No sibling has such a header: the musespark entries at nvidia-master.yaml:6978 and 7078, and the sibling agg-variants.yaml, all have none. Before this change neither file had these headers. After merge they hold provenance that goes stale on the next image bump.
Verification: Severity: nit. Both blocks break the repository's mandatory comment rule in AGENTS.md:54-58, which bans header comments describing a recipe, topology or config, and history of where code came from. agg-mk-variants.yaml:3-5 describes the recipe and topology. nvidia-master.yaml:7048-7052 describes the config and records provenance. Nothing at runtime breaks.
sgl-muse-mega's draft build reads config.json and model.safetensors from a local --speculative-draft-model-path and refuses a Hub id, and SGLang 5884e4fec does not resolve Hub ids to local snapshots, so the server would fail at startup. Point the draft at its pinned snapshot in the mounted Hub cache, and keep the megakernel's JIT build cache on node-local disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new megakernel key shares model, hardware, framework and precision with the dflash-agg and dflash-disagg keys, so it forms one dashboard curve with them and the newest run replaces that curve. List both sibling keys so the sweep republishes every point instead of leaving only the c1 megakernel point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=44425826871 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=44425826871 |
…ontext
At mem-fraction-static 0.6 the megakernel's ~60 GB per GPU left the
post-capture KV pool at 541,440 tokens, so AgentX turns past ~541K
input tokens were rejected ("Input length exceeds the maximum allowed
length") and the c1 canary hit the 10% failed-request limit. 0.7 lets
the pool reach its 1,086,592-token sizing with ~80 GB per GPU still free.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] sgl-project/sglang#50289 has been merged. This PR's AgentX config ( This PR has no authorized |
…ernel-c1 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#50766 has been merged. This PR's AgentX config ( This PR has no authorized |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] @Fridge003 Heads-up: I'm actively babysitting this PR's pipeline (currently run 44232639011). Please don't cancel or rerun the pipelines yourself while the babysit is in progress. I'll handle reruns for infrastructure failures, and parallel cancels or reruns collide with that and push our jobs back in the GB300 queue. Current status: the 1P2D c160 point failed for an infrastructure reason (a corrupted container image on one GB300 node made the container fail to start), and fail-fast cancelled the remaining throughput points. Once the eval phase finishes I'll rerun the failed jobs; they keep this run's priority (3.44). If you need to push a change or want to take over, please comment here first. |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#51504 has been merged. This PR's AgentX config ( This PR has no authorized |
Adds a single Muse Spark Open GB300 AgentX point at concurrency 1 that runs the sgl-muse-mega NVIDIA megakernel (
--enable-nvidia-megakernel). The plugin fuses each batch-1 decode step (DFlash2 draft, target verify and accept) into one CUDA graph. It needs its own image, so the point goes in a new config key,musespark-fp4-gb300-dynamo-sglang-agentic-dflash-mk-agg, with a regular changelog entry.The new key shares model, hardware, framework and precision with
musespark-fp4-gb300-dynamo-sglang-agentic-dflash-aggand-disagg, so the three keys form one dashboard curve, and the newest run replaces that whole curve. The changelog entry therefore lists all three keys and the sweep reruns the full Muse Spark GB300 Dynamo + SGLang curve (15 points). The sibling keys' image, recipes and points are unchanged. Other Muse Spark curves are not touched.lmsysorg/sglang:dev-0775fc6-20261112-cu13-arm64@sha256:a200999b…(SGLang5884e4fec, sgl-muse-mega0775fc6, ai-dynamo1.7.0.dev20261026from the base nightly).trtllm_mhaattention, 128-token pages.SGLANG_MUSE_MEGA_TREE=0). Synthetic acceptance uses the committed golden curve: 7 drafts, AL 4.05. sgl-muse-megaf009237and later forceSGLANG_SIMULATE_ACC_LENon the fused path. The plugin refuses the token tree under simulation, and the curve covers no 16-token block.Deviations from the other Muse Spark points, for review:
temperature: 0on the server and inAIPERF_EXTRA_INPUTS). Under simulated acceptance the plugin runs only greedy batch-1 decode steps; sampled requests fall back to SGLang's own step. The existing Muse Spark curves servetemperature 1.0 / top_k 64. Acceptance is forced from the golden curve either way.infx.srt_slurm.spec_identityreportskv_cache_dtypebf16 vs the curve's fp8 as a mismatch; the AL value itself matches the golden curve.--context-length 1048576. The megakernel requires an explicit context length. Plugin0775fc6serves contexts up to SGLang's rope rows (1M), so the uncapped AgentX traces need no client cap.config.jsonandmodel.safetensorsfrom a local--speculative-draft-model-pathand refuses a Hub id; SGLang5884e4fecdoes not resolve Hub ids to local snapshots. The recipe points at the same pinned draft (RadixArk/Muse-Spark-Open-DFlash2at46c90f41) through its snapshot under the mounted/hf_hub_cache.dev-0775fc6; the agg and disagg points stay onnightly-dev-cu13-20261103-797af717, as in [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang: 20261111 nightly and a DEP32 mid-curve point / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang:升级到 20261111 nightly 并加入 DEP32 中段点位 #4997. The existing agg TP8 c1 point (sampled, FP8 KV, no megakernel) stays on the curve for comparison.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied./use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
新增一个 Muse Spark Open GB300 AgentX 并发 1 测试点,使用 sgl-muse-mega NVIDIA megakernel(
--enable-nvidia-megakernel)。该插件把每个 batch-1 decode step(DFlash2 draft、target verify 与 accept)融合进一个 CUDA graph。由于需要单独的 image,该点放在新的 config keymusespark-fp4-gb300-dynamo-sglang-agentic-dflash-mk-agg中,使用普通 changelog 条目。新 key 与
musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg和-disagg的 model、hardware、framework、precision 都相同,三者构成同一条 dashboard 曲线,最新一次运行会替换整条曲线。因此 changelog 条目列出全部三个 key,sweep 重跑整条 Muse Spark GB300 Dynamo + SGLang 曲线(15 个点)。两个已有 key 的 image、recipe 和点都不变;其他 Muse Spark 曲线不受影响。lmsysorg/sglang:dev-0775fc6-20261112-cu13-arm64@sha256:a200999b…(SGLang5884e4fec、sgl-muse-mega0775fc6、base nightly 自带的 ai-dynamo1.7.0.dev20261026)。trtllm_mhaattention,page size 128。SGLANG_MUSE_MEGA_TREE=0)。合成接受率使用已提交的 golden 曲线:7 个 draft,AL 4.05。sgl-muse-megaf009237起在 fused 路径上强制SGLANG_SIMULATE_ACC_LEN;插件在模拟模式下拒绝 token tree,golden 曲线也没有 16-token block 的数据。与其他 Muse Spark 点的差异,请 reviewer 关注:
AIPERF_EXTRA_INPUTS均设temperature: 0)。模拟接受率下,插件只对 greedy batch-1 decode step 走 megakernel,sampled 请求回落到 SGLang 自身的 step。现有 Muse Spark 曲线使用temperature 1.0 / top_k 64;两种情况下接受率都由 golden 曲线强制。infx.srt_slurm.spec_identity报告kv_cache_dtypebf16 与曲线的 fp8 不一致;AL 数值本身与 golden 曲线一致。--context-length 1048576。 megakernel 要求显式指定 context length。插件0775fc6支持到 SGLang rope rows 的长度(1M),因此不封顶的 AgentX trace 无需在客户端截断。--speculative-draft-model-path读取config.json与model.safetensors,不接受 Hub id;SGLang5884e4fec也不会把 Hub id 解析为本地 snapshot。recipe 通过挂载的/hf_hub_cache下的 snapshot 路径指向同一个钉住版本的 draft(RadixArk/Muse-Spark-Open-DFlash2,revision46c90f41)。dev-0775fc6,agg 与 disagg 点仍使用nightly-dev-cu13-20261103-797af717,与 [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang: 20261111 nightly and a DEP32 mid-curve point / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang:升级到 20261111 nightly 并加入 DEP32 中段点位 #4997 的做法一致。已有的 agg TP8 c1 点(sampled、FP8 KV、无 megakernel)保留在曲线上用于对比。AI model disclosure
Prepared with Claude Code using Claude Opus 5.5 (
claude-opus-5-5): image and plugin inspection, config, recipe and changelog authoring, local matrix generation and spec-identity audit, and this description. Later pipeline fixes (KV pool sizing, merges of main, changelog wording) and the checklist were also made with Claude Opus 5.5 (claude-opus-5-5). No delegated agents.🤖 Generated with Claude Code