Repository navigation
[AMD] MiniMax-M3.1-Flash MI355X vLLM: latest nightly image and merged optimization flags / [AMD] MiniMax-M3.1-Flash MI355X vLLM：最新 nightly 镜像与已合入优化的开关 - #4866
Conversation
… optimization flags Pin vllm/vllm-openai-rocm nightly-4fab5a07 (2026-11-06) by digest and enable the MiniMax-M3.1 ROCm optimizations merged since the day-zero image: drafter monokernel, parallel decode top-k, Rust frontend, INT4 quick all-reduce, skinny GEMM off, and breakable CUDA graphs at c16/c32. DBO is removed from c16-c64. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Fangzhou-Ai
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # c16-c32: DSpark 7 on the NVFP4 KV cache. Breakable CUDA graphs capture the | ||
| # prefill and mixed steps too. The capture list (every multiple of 8 up to 256, | ||
| # then 384-8192; a coarse list makes decode about 10% slower) is a deprecated | ||
| # alias on the V2 runner that sets both the prefill token sizes and the decode | ||
| # request sizes (1-32 at c16); the new per-kind fields would change the decode | ||
| # graphs. Admission is 2x CONC. |
There was a problem hiding this comment.
🟡 nit (optional): The rewritten c16-c32 section comment is a header describing an override, which AGENTS.md's comment rules forbid. It also cites a past result ("a coarse list makes decode about 10% slower"). Separately, line 90 is left as an orphaned fragment: "8192-token prefill steps on the V1 runner." Fix: cut the header to a one-line reason on the compilation-config line, and delete or complete the line 90 fragment.
Why this was flagged
AGENTS.md:56-58 says: no header comments describing an override or section, no past results, and only short technical reasons tied to a specific line. The diff rewrites the comment above zip_override_tp4_dspark7 (agentic.yaml:111-116) into a six-line override description. That description includes the measured "about 10% slower" result and narrates the alias behaviour. The DBO edit at agentic.yaml:90 left the sentence fragment "8192-token prefill steps on the V1 runner.", which no longer explains anything. This is not a runtime issue; it is a convention violation the diff introduces.
Verification: AGENTS.md:54-58 says "No header comments describing a file, topology, config, recipe, override, or section" and "No history: ... or past results". The diff rewrites the section comment at agentic.yaml:111-116 into a six-line description of the override. At agentic.yaml:90 it leaves a sentence fragment with no subject. It does not change how anything runs.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42224368840 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42224368840 |
…educeInt4.compile Workers failed in init_device at every point (first CI run, c1). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e opt-in optimizations Update the mi355x_agentx preset to the InferenceX lane that passed on the latest nightly (SemiAnalysisAI/InferenceX#4866): pin vllm-openai-rocm nightly-4fab5a07, enable parallel index top-k and the Rust frontend, turn the skinny GEMM off, and run concurrency 16 and 32 on breakable CUDA graphs. Dual-batch overlap is removed from the 16-64 points. Refresh the runbook table and notes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
|
/use 42224368840 |
|
@chunfangamd staged run 42224368840. This run remains available across future |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42224368840
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/42224368840
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
Draft data types: no change. The checkpoint is unchanged: amd/MiniMax-M3.1-Flash-Quark-MXFP4 (HF 8f274761).
- Draft model. The DSpark draft in
dspark/is a standalone 5-layerDSparkMiniMaxDraftModelwithquant_method: mxfp8andweight_block_size: [1, 32].- The attention q/k/v/o projections and the dense FFN gate/up/down are MXFP8: FP8 E4M3 with 1×32 UE8M0 scales and dynamic activations.
- fc, the Markov head, the confidence head and the norms are BF16.
- Loading. At the image's vLLM
4fab5a07(0.33.0rc1.dev910+g4fab5a074),_get_minimax_m3_dspark_draftstill resolves the draft to thatdspark/subfolder.DFlashQwen3Modelbuilds the draft's quant config from it throughget_draft_quant_config, not from the target's Quark config. - Recipe. It sets no draft
quantizationordtype.speculative-configkeeps the FP8 draft KV cache andTRITON_ATTN, as on base. - The one draft-side change is
VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER=1.- It runs seven-token draft steps of up to 10 requests (c1–c32) as fused FlyDSL kernels. c64's three-token draft refuses them (
drafter kernels off: 7 speculative tokens required) and runs the stock layers. - The kernels keep the stock precisions:
- MXFP8 GEMVs on the stored MXFP8 weights;
- activations quantized to MXFP8 before QKV, o_proj, gate/up and down, as the stock path's
kMxfp8Dynamicdoes; - FP8 attention on the FP8 draft KV with an FP8 query, as the stock FP8-KV path does;
- BF16 norms, residuals and partial sums.
- It runs seven-token draft steps of up to 10 requests (c1–c32) as fused FlyDSL kernels. c64's three-token draft refuses them (
- Acceptance. The AL gate on the eval servers measured the real draft at 4.0467 (c8) and 4.0234 (c32) against the 4.04 golden value, and at 2.8576 (c64) against 2.88.
Sign-off evidence (PR review checklist)
Swept at ce4079c; assessed at the head f54e244. The later commits 19a9961 and f54e244 only merge main. The recipe, the master entry and this PR's changelog entry are unchanged, infx.matrix.plan generates the same 10 entries (seven points and three evals), and no main commit in between touches the launch, benchmark or eval code.
| Item | Status | Evidence |
|---|---|---|
| PR validation | ✅ | Run 42224368840 on ce4079c: all seven points passed (TP4 c1, c2, c4, c8, c16 and c32 without offload; c64 with dram / vllm-simple). c16 ran twice with the same effective configuration on clean GPUs: drafter kernels on, breakable graphs, Rust frontend and synthetic AL 4.04. Attempt 1 (mia1-p01-g11) measured 26,846 tok/s/GPU at 12.72 s p90 E2EL with a 5.64 ms median TPOT. Attempt 2 (mia1-p01-g16) measured 28,686 tok/s/GPU at 8.40 s with 3.19 ms; the cause of the gap is not identified. This sign-off and the upstream runbook use attempt 2, and results_bmk keeps both rows. |
| Evals | ✅ | Same run, agentic eval, gsm8k (1319 questions), one eval per engine-env group: c8 strict 0.9606 / flexible 0.9598, c32 0.9629 / 0.9621, c64 0.9598 / 0.9591. All are within 2σ of the exact MI355X vLLM cohort (0.9568–0.9666; worst gap 0.0068 vs 0.0146). The eval server logs quoted under Eval server-log evidence below show the changed paths ran: mono decode and drafter kernels at c8, and drafter kernels and breakable CUDA graphs at c32. All three servers served the GSM8K requests through the Rust frontend, with VLLM_ROCM_PARALLEL_INDEX_TOPK=1 and VLLM_ROCM_USE_SKINNY_GEMM=0 in the worker env; c8's FP8 indexer bypasses the parallel top-k. |
| Chat templates | ✅ | AgentX replays the traces through /v1/chat/completions, so the served chat template applies. The Rust frontend applies the model's chat template and has a minimax_m3 tool parser; it treats --enable-auto-tool-choice as a no-op. Prompts did not shrink: server prompt tokens per request are 162.6K at c8 and 178.7K at c32, against 158.3K and 173.2K in run 41003224750. |
| Draft precision | ✅ | Above: the DSpark draft runs as shipped (MXFP8 projections and FFN; BF16 fc, heads and norms) with an FP8 draft KV cache, and the drafter kernels keep those precisions. There is no draft quantization or dtype override. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE does not apply (vLLM path). |
| Golden AL | ✅ | Throughput jobs inject synthetic acceptance 4.04 at k=7 (c1–c32) and 2.88 at k=3 (c64): the c16 config sets synthetic_acceptance_length: 4.04, and the c64 server reports mean acceptance length 2.88. On current main, signoff_facts golden-al selects minimax-m3.1-flash__dspark__native__probabilistic-block.yaml (thinking_on, probabilistic drafting, block verification) and matches both values. |
| MODELS.md | ✅ | minimaxm31flash (released 2026-10-23) agentic coding (vLLM DSpark) is active. |
| Architecture hacks | ✅ | No --hf-overrides or layer skipping. The parallel index top-k orders blocks by the same score-then-index key as the serial FlyDSL top-k, and vLLM tests it against that path, so it selects the same blocks. The drafter kernels, breakable CUDA graphs, the Rust frontend and the DBO removal change only how the same computation runs. |
| Integer all-reduce | ✅ | Disabled. ce4079c dropped VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4, and neither the recipe nor the image sets *QUICK_REDUCE_QUANTIZATION. vLLM 4fab5a07 defaults it to NONE, and the servers use AITER_CUSTOM and PYNCCL for the TP all-reduce. |
| Dynamo TRT-LLM metrics | N/A | Direct vLLM serving; no Dynamo. |
| Upstream image / engine-first | ✅ | vllm/vllm-openai-rocm:nightly-4fab5a07475822c6eaa971dceb2e2220d0446808@sha256:6f126f27…, from the upstream vLLM org. vLLM is the engine itself, so the ordering rule does not apply. |
| Main provenance | ✅ | The image's vLLM 4fab5a07 is on vLLM main (0 ahead, 81 behind). It replaces the day-zero image that #4671 ran under the new-model window. |
| Upstream recipe | ✅ | vllm-project/vllm#66188 merged on 2026-11-07 (80e0cd68). Its mi355x_agentx preset on the published recipe cites this PR at ce4079c and pins the same image and digest on the mxfp4 (Quark) variant at TP4. It sets the same values per point as this recipe: c1–c8 DSpark 7 with mono decode (FP8 KV and indexer caches, FULL_DECODE_ONLY, shuffled KV layout, 0.85, 8192-token steps); c16/c32 DSpark 7 on the NVFP4 KV cache with breakable CUDA graphs (FULL_AND_PIECEWISE, the same capture list, V2 runner, DEBUG_HIP_DYNAMIC_QUEUES=0) at 0.85 with 32768-token steps; c64 DSpark 3 at 0.90 with SimpleCPUOffloadConnector (1349 GB eager) and PYTHONHASHSEED=42. Every point has --max-num-seqs at 2× concurrency, block verification, prefix caching, VLLM_HTTP_TIMEOUT_KEEP_ALIVE=900 and the four new switches (VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER, VLLM_ROCM_PARALLEL_INDEX_TOPK, VLLM_USE_RUST_FRONTEND, VLLM_ROCM_USE_SKINNY_GEMM=0). Shared settings come from the mxfp4 variant (--moe-backend aiter, no shared-expert fusion) and the AMD override (NVFP4 KV, indexer_decode_cp). Excluded as InferenceX-only or inert: served-model-name and max-model-len (the model id and its default 1M context); trust-remote-code (vLLM serves MiniMaxM3SparseForConditionalGeneration natively); disable-uvicorn-access-log, PYTHONNOUSERSITE, PYTHONUNBUFFERED, VLLM_ENGINE_READY_TIMEOUT_S and the benchmark env; and c64's VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1, which vLLM 4fab5a07 never reads (the server logs it as an unknown variable). |
| Engine patches | ✅ | No patches, setup scripts, overlays or waivers; the pinned image runs as shipped. |
| Append-only | N/A | The changelog block does not set append-only: true. |
| Re-tune deltas | ℹ️ | Against the published run 41003224750 (day-zero image, DBO at c16–c64), throughput per GPU changes by +12.6%, +9.7%, +2.2%, +4.5%, +5.2%, +7.1% and +20.0% at c1, c2, c4, c8, c16, c32 and c64. Mean interactivity changes by +2.9%, +6.4%, +5.1%, +16.1%, +23.4%, +18.1% and +28.5%. With c16's first attempt, c16 is −1.6% and −20.0%. |
| Pareto counts | One curve: minimaxm31flash / agentic-coding / MI355X vLLM / fp4 / p90, run 42224368840 results_bmk. pareto_coverage gives 7 measured, 0 invalid and 4 on the frontier: c8 (16,104 tok/s/GPU, 7.00 s), c16 (28,686, 8.40 s), c32 (58,024, 13.22 s) and c64 (90,495, 27.17 s). c1, c2 and c4 are dominated by c8, whose p90 E2EL is now below c4's. c16's first attempt gives the same count. With fewer than 5, core maintainer review and a recorded admin bypass are needed before merge. cc @functionstackx |
Eval server-log evidence (Check 2)
Quoted verbatim from the eval_server_logs_* artifacts of run 42224368840; each file is inside the artifact's srt-single-node-logs.tar.gz.
-
The drafter runner is built at the first eligible draft step outside CUDA-graph capture, which here is the startup warm-up, so the graphs captured afterwards include its kernels.
-
It stays on: neither the c8 log nor the c32 log shows a later
drafter kernels offwarning. Every draft step of up to 10 requests (seven query rows each) then runs on the kernels. -
The
RustFrontendstats lines show the Rust frontend serving the GSM8K requests. -
Uncompressed, byte-identical copies of the c8 and c32 worker logs and fingerprints are attached in this comment, so the quoted lines can be read without opening the archives:
- c8:
c8_eval_server_mia1-p01-g37_agg_w0.log(sha256a7ecdd917bfcaedd72917e13b1507f302495c10d56c05814527795431279cbdf) andc8_eval_fingerprint_agg_w0.json(sha2566c0bd01252494fe7a18e38ce927c0103bcc73e42ec41f294a1687ccf30ba4cf9). - c32:
c32_eval_server_mia1-p01-g16_agg_w0.log(sha2566d8fa87dac3547cfc29c0cc9f7310b49a59f79dcba3d4987c5cff7a878b93451) andc32_eval_fingerprint_agg_w0.json(sha256a52a5ced91844d9ddfa0b5232b1e63409253b3d9fbaf4adb05e8d893192d64ce). SHA256SUMS.txttraces each file to its artifact. Each artifact zip hashes to its GitHub APIdigest(artifact 13526599200:sha256:a47e70382e9946828536432f732fd8fcfed7418f242939c3f818ec89a0b4b87d; artifact 13524284855:sha256:aed6a1031a1117fe207e41eda183e59d3407fc6d3203c0f0c514ac980c4c419c), and thesrt-single-node-logs.tar.gzinside hashes to7061b523bf0765f9e7c4e3754861c647a4d1943de94f4e0c9d30ccc6ddddc14f(c8) and93e941d4e9e2d53f866deb856960687745ddaca65adc7de1e47b67744c35023a(c32).
- c8:
-
c8: artifact
eval_server_logs_minimaxm31flash_tp4_conc8_kvnone_spec-mtp_fp4_vllm_tp4-pp1-dcp1-pcp1-ep1-dpafalse_disagg-false_spec-3d7d19171a8f415fc5f5_lm-eval, filelogs/mia1-p01-g37_agg_w0.outL27: (MainProcess pid=828225) INFO 11-06 17:47:03 [utils.py:372] Launching Rust frontend: /usr/local/lib/python3.12/dist-packages/vllm/vllm-rs frontend ... L424: (Worker_TP0 pid=831568) INFO 11-06 17:52:36 [dispatch.py:90] MiniMax-M3 DSpark drafter kernels on [TP1-TP3: L425-L427] L432: (Worker_TP0 pid=831568) INFO 11-06 17:52:42 [dispatch.py:174] MiniMax-M3 mono decode on: shared expert separate MXFP8 (0.77 GiB tiled copy per rank) [TP1-TP3: L433-L435] L683: (RustFrontend pid=831493) INFO 11-06 17:55:10 [log_stats.rs:300] Avg prompt tput: 411.7 toks/s, Avg generation tput: 595.2 toks/s, Reqs Running: 16, Waiting: 48, GPU KV cache used: 0.1%, Prefix cache hit rate: 32.0% [first non-zero stats] L3935: (RustFrontend pid=831493) INFO 11-06 18:00:20 [log_stats.rs:300] Avg prompt tput: 6102.1 toks/s, Avg generation tput: 1004.4 toks/s, Reqs Running: 0, Waiting: 0, GPU KV cache used: 0.0%, Prefix cache hit rate: 13.0% [last stats before shutdown]logs/fingerprint_agg_w0.json: L43"VLLM_ROCM_MINIMAX_M3_MONO": "1", L44"VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER": "1", L45"VLLM_ROCM_PARALLEL_INDEX_TOPK": "1", L50"VLLM_ROCM_USE_SKINNY_GEMM": "0", L52"VLLM_USE_RUST_FRONTEND": "1" -
c32: artifact
eval_server_logs_minimaxm31flash_tp4_conc32_kvnone_spec-mtp_fp4_vllm_tp4-pp1-dcp1-pcp1-ep1-dpafalse_disagg-false_spe-bed462d9dd7d18709199_lm-eval, filelogs/mia1-p01-g16_agg_w0.outL27: (MainProcess pid=1010486) INFO 11-06 17:17:32 [utils.py:372] Launching Rust frontend: /usr/local/lib/python3.12/dist-packages/vllm/vllm-rs frontend ... L352: (Worker_TP0 pid=1013684) INFO 11-06 17:22:44 [dispatch.py:90] MiniMax-M3 DSpark drafter kernels on [TP1-TP3: L351, L353, L354] L407: (Worker_TP0 pid=1013684) INFO 11-06 17:23:03 [breakable_cudagraph.py:303] Breakable CUDA graph enabled L765: (Worker_TP0 pid=1013684) INFO 11-06 17:23:45 [model_runner.py:1226] Graph capturing finished in 43 secs, took 9.99 GiB [TP1-TP3: L756, L759, L762] L866: (RustFrontend pid=1013608) INFO 11-06 17:25:38 [log_stats.rs:300] Avg prompt tput: 2194.0 toks/s, Avg generation tput: 4230.2 toks/s, Reqs Running: 64, Waiting: 0, GPU KV cache used: 0.5%, Prefix cache hit rate: 30.9% [first non-zero stats] L3758: (RustFrontend pid=1013608) INFO 11-06 17:29:58 [log_stats.rs:300] Avg prompt tput: 0.0 toks/s, Avg generation tput: 9.7 toks/s, Reqs Running: 0, Waiting: 0, GPU KV cache used: 0.0%, Prefix cache hit rate: 0.0% [last stats before shutdown]logs/fingerprint_agg_w0.json: L43"VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER": "1", L44"VLLM_ROCM_PARALLEL_INDEX_TOPK": "1", L48"VLLM_ROCM_USE_SKINNY_GEMM": "0", L50"VLLM_USE_BREAKABLE_CUDAGRAPH": "1", L51"VLLM_USE_RUST_FRONTEND": "1" -
c64: artifact
eval_server_logs_minimaxm31flash_tp4_conc64_kvdram-vllm-simple_spec-mtp_fp4_vllm_tp4-pp1-dcp1-pcp1-ep1-dpafalse_disa-8f774035eb494bc32911_lm-eval, filelogs/mia1-p01-g09_agg_w0.outL27: (MainProcess pid=1332952) INFO 11-06 17:51:32 [utils.py:372] Launching Rust frontend: /usr/local/lib/python3.12/dist-packages/vllm/vllm-rs frontend ... L232: (Worker_TP0 pid=1338875) INFO 11-06 17:56:00 [dispatch.py:65] MiniMax-M3 DSpark drafter kernels off: 7 speculative tokens required [TP1-TP3: L230, L241, L245; expected for the 3-token draft] L1061: (RustFrontend pid=1338796) INFO 11-06 18:00:06 [log_stats.rs:300] Avg prompt tput: 2150.1 toks/s, Avg generation tput: 3752.1 toks/s, Reqs Running: 64, Waiting: 0, GPU KV cache used: 0.4%, Prefix cache hit rate: 30.6%, External prefix cache hit rate: 0.0% [first non-zero stats]logs/fingerprint_agg_w0.json: L43"VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER": "1", L45"VLLM_ROCM_PARALLEL_INDEX_TOPK": "1", L49"VLLM_ROCM_USE_SKINNY_GEMM": "0", L51"VLLM_USE_RUST_FRONTEND": "1"
Signed: @chunfangamd
|
|
Uncompressed copies of the c8 and c32 eval server logs from run 42224368840, for the verifier (Check 2). They are byte-identical to the tar members (only the file names changed); SHA256SUMS.txt traces each file to its artifact (zip sha256 = GitHub API digest) and the srt-single-node-logs.tar.gz inside it. c8_eval_fingerprint_agg_w0.json |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
Doesn't minimax use nvfp4 kvcache? I see an PR in upstream vllm working on supporting nvfp4 kvcache for gfx950
Do u have tb4.0 or DeepSWE benchmark on how mxfp4 vsnvfpR kvcache would affect accuracy?
| - "At concurrency 80 the overall prefix cache hit rate is flat: 93.5% of prompt tokens on the previous image and policy (run 36743632401) against 94.0% here (run 39763900198). What moves is the tier split -- the HiCache host tier serves 49.1% of prompt tokens instead of 43.3%, and the GPU tier 44.9% instead of 50.2%. Per-GPU throughput goes from 59,233 to 60,839 tok/s (+2.7%), p90 end-to-end latency from 50.9 s to 47.9 s, and mean TTFT from 3.12 s to 2.68 s." | ||
| - "That concurrency 80 comparison is not a single-variable A/B: the 36743632401 baseline ran hicache-size 400 on lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260929, while this point runs hicache-size 374 on the v0.5.22 image, so the image, the write policy and the host pool size all differ." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4374 | ||
| - "Keep the batch-8 cap on running requests and decode gr |
There was a problem hiding this comment.
thanks! hi @chunfangamd can you help me to do that, I don't want to accidentally trigger another repeated perf sweep here.
|
here is the mono kernel for MI355 on the OG ckpt vllm-project/vllm#65638 |
Hi @functionstackx we are not using mxfp4 kv cache anywhere, we cast some of them back to fp8 under some cases, so I see no potential precision regression here. |
|
hi @Fangzhou-Ai ur right, i just dismissed my request change |
Restore main's perf-changelog.yaml, which the previous merge truncated, and re-append this PR's entry from ce4079c.
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 8, rawBenchmarkRows 15, coveredByPerJobFiles 8, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 3957, gpuMetricSamplesNew 126728, powerPublicationPoints 7, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
MiniMax-M3.1-Flash on MI355X (vLLM, TP4, Quark MXFP4) still runs the day-zero image
minimax-m31-flash, which predates the ROCm work merged since. This PR moves it to the latest nightly and turns on the optimizations that need an opt-in switch, so the recipe actually uses them.What changes (
minimaxm31flash-fp4-mi355x-vllm-agentic-dspark, all 7 points regenerate)vllm/vllm-openai-rocm:nightly-4fab5a07475822c6eaa971dceb2e2220d0446808@sha256:6f126f27…(vLLM main of 2026-11-05). It contains decode CP (#64448), qkv/FFN norm+quant fusions (#63637, #65478), four-wave decode top-k (#65087), local DSpark 7 indexer scoring (#65432), per-XCD sparse-decode splits (#65481), the MXFP8 tile retune (#65530) and the drafter monokernel (#65386), so those apply automatically.VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER=1(#65386),VLLM_ROCM_PARALLEL_INDEX_TOPK=1(#65267, NVFP4 indexer points),VLLM_USE_RUST_FRONTEND=1,VLLM_ROCM_USE_SKINNY_GEMM=0(removes the draft LM head's 5-request skinny-GEMM cliff, 0.28 vs 0.05 ms per call).VLLM_USE_BREAKABLE_CUDAGRAPH=1,FULL_AND_PIECEWISE, multiple-of-8 capture list).max-num-seqs, prefill budgets.INT4 quick all-reduce is not enabled here, and is a problem for #4485 on this image. The first CI run of this PR failed at worker startup with
AttributeError: 'QuickAllReduceInt4' object has no attribute 'compile': the nightly's vLLM (#59478) calls.compileon AITER's FlyDSL INT4 object, and the AITER built into the image (0.1.28.post1) does not have it. Any recipe on this image that setsVLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4will fail the same way (#4485 included, if its image is bumped to a nightly of this age).Validation
Run 42224368840 on
ce4079cpassed all seven points and the c8, c32 and c64 GSM8K evals (strict 0.9606, 0.9629 and 0.9598), and/use 42224368840is posted. The concurrency 16 job ran twice: the second attempt (28,686 tok/s/GPU at 8.40 s p90 E2EL) is the one used, and the first (26,846 at 12.72 s) stays inresults_bmk. The first CI run (c1, on6aec58e) failed at startup on the INT4 setting above, which this revision removes. Before opening, I started the c16 variant (rendered from this recipe) on 4x MI355X with the nightly-commit vLLM code: healthy, drafter kernels on, 45 prefill + 32 decode graphs captured, no errors, mean acceptance length 4.5-4.65, GSM8K 0.945 (200 questions, c16), 22K-token prefill served. Not covered locally: the Rust frontend (no matching binary), the c1-c8 mono points (this container's AITER 0.1.27 fails the mono kernel; the image builds AITER 0.1.28.post1) and c64. The local test also ran with INT4 on and passed, because the local AITER has the method the image lacks, so a local pass did not predict the CI failure.Validation details
dspark/draft keeps its MXFP8 config and an FP8 draft KV cache. The drafter kernels replace the draft layers' MXFP8 linear, norm and attention ops; vLLM'stests/models/minimax_m3/test_dspark_drafter.pyholds them to the stock layers' own numerical spread. At c64 (3 draft tokens) they refuse and the stock layers run.中文
MiniMax-M3.1-Flash 在 MI355X 上的 vLLM 配方仍使用首日镜像
minimax-m31-flash，不含此后合入的 ROCm 优化。本 PR 将镜像升级到最新 nightly（按 digest 固定），并打开需要显式开关的优化。nightly-4fab5a07475822c6eaa971dceb2e2220d0446808，已包含 decode CP、norm+quant 融合、四波 decode top-k、按 XCD 的稀疏解码切分、MXFP8 tile 重调、drafter monokernel 等。VLLM_ROCM_MINIMAX_M3_MONO_DRAFTER=1、VLLM_ROCM_PARALLEL_INDEX_TOPK=1、VLLM_USE_RUST_FRONTEND=1、VLLM_ROCM_USE_SKINNY_GEMM=0。QuickAllReduceInt4.compile，worker 启动即失败（首次 CI 的 c1 即因此失败）；Enable INT4 QuickReduce for MiniMax M3.1 MI355X vLLM / 为 MiniMax M3.1 MI355X vLLM 启用 INT4 QuickReduce #4485 若升级到同龄 nightly 也会遇到同样问题。ce4079c）七个点与 c8、c32、c64 的 GSM8K 评测全部通过（strict 0.9606、0.9629、0.9598），/use 42224368840已发布；c16 运行了两次，使用第 2 次结果（28,686 tok/s/GPU，p90 E2EL 8.40 s），第 1 次（26,846，12.72 s）保留在results_bmk中。开 PR 前已在本地用 nightly 提交的代码验证 c16 变体可启动、GSM8K 0.945（200 题）。Rust 前端、c1-c8 mono 点与 c64 未能在本地验证。AI model disclosure
Claude Sonnet 5.5 (
claude-sonnet-5-5[1m]) through Claude Code prepared the recipe, config and changelog edits, ran the local checks and the c16 smoke test, and wrote this description. No delegated models contributed. Claude Opus 5.5 (Cursor agent) later updated the Validation section, at the reviewer's request, to cite run 42224368840.Type of change