Repository navigation
feat(agentx): MiniMax-M3.1-Flash FP4 MI355X ATOM AgentX, Quark MXFP4 served as shipped / feat(agentx):新增 MiniMax-M3.1-Flash FP4 MI355X ATOM AgentX,Quark MXFP4 按原样服务 - #4860
Conversation
d311cab to
f2884f6
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Two configuration comments redundantly restate settings and need removal or trimming to comply with the repository’s comment requirements.
2 open issues
What changed in this PR
Adds MiniMax-M3.1-Flash FP4 AgentX benchmarking on MI355X with ATOM, serving the Quark checkpoint and bundled DSpark draft at their shipped precisions.
Changes:
- Registers seven TP4 concurrency points, with LMCache CPU offloading at C16–C48.
- Adds a pinned-image recipe with FP8 caches, concurrency-specific decode graphs, and prefill delays at C1–C16.
- Appends a performance-changelog entry for sweep collection.
| File | Description |
|---|---|
inferencex-e2e/perf-changelog.yaml |
Records the new benchmark configuration and tuning. |
inferencex-e2e/configs/amd-master.yaml |
Registers the model, image, runner, and concurrency tiers. |
inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/minimaxm31flash/atom/mi355x-fp4-mtp/agentic.yaml |
Defines ATOM serving, DSpark speculation, and LMCache variants. |
Review effort: Balanced
| # One variant per point. Admission is 2x CONC (at least 2); decode graphs cover | ||
| # every batch size up to 8, then steps of 2 / 4 / 8 / 16 up to the admission. | ||
| # C1-C16 run the TP-local prefill delayer: after a prefill, no new prefill is | ||
| # admitted for 20000 scheduler passes while requests decode. C16-C48 add the | ||
| # LMCache CPU tier at 200 GiB per rank (800 GiB, 859 GB per server), since the | ||
| # FP8 KV and index pool alone (about 7.3M tokens) is smaller than the AgentX | ||
| # working set at C48. |
| # K7) from infx/srt_slurm/synthetic_acceptance.py; evals keep real | ||
| # acceptance. | ||
| method: dspark | ||
| draft-model: /model/dspark |
There was a problem hiding this comment.
🟡 (optional) Results from this config will record the draft model as the container path /model/dspark, not native like every other bundled-draft config. The cause is the explicit draft-model: /model/dspark at agentic.yaml:42. spec_parameters copies it as the draft model, and speculation_facts only rewrites HF cache paths or an empty value to native. Fix: drop draft-model if ATOM finds the checkpoint's dspark/ itself, as the dsv41flash ATOM recipe does. Otherwise make speculation_facts report native for a draft inside the served checkpoint.
Why this was flagged
For ATOM, spec_parameters (infx/srt_slurm/synthetic_acceptance.py:82) sets model to args.get("draft-model", ""), which is /model/dspark here. speculation_facts (infx/srt_slurm/spec_identity.py:48-54) returns native only when that string is empty. A path matching models--<org>--<name> becomes a repo name. /model/dspark matches neither, so it is passed through unchanged. bind_speculation (infx/srt_slurm/workload.py:479-481) writes it to SPEC_DRAFT_MODEL. speculation_fields (infx/results/metadata.py:45-48) stores that as spec_draft_model in every throughput and eval result for minimaxm31flash-fp4-mi355x-atom-agentic-dspark. The draft is the checkpoint's own bundled dspark/, the same draft the SGLang and vLLM M3.1-Flash configs record as native. No check rejects the path, because compare does not look at the draft field.
Verification: nit. The recipe sets draft-model: /model/dspark (agentic.yaml:42). speculation_facts (infx/srt_slurm/spec_identity.py:48-54) rewrites only HF cache paths (models--<org>--<name>), and returns native only when the value is empty (model or "native"). /model/dspark matches neither, so it passes through unchanged. Curve selection is not affected.
| # MiniMax-M3.1-Flash AgentX on MI355X with ATOM: AMD's Quark MXFP4 export served as | ||
| # shipped (no online quantization, no shared-expert fusion, no quick-reduce | ||
| # quantization) at TP4 with the bundled seven-token DSpark draft, FP8 KV and FP8 | ||
| # index cache. C1-C16 run the prefill delayer; C16-C48 add the LMCache CPU tier. | ||
| # Throughput runs use the minimax-m3.1-flash__dspark__native__probabilistic-block | ||
| # golden AL. |
There was a problem hiding this comment.
🟡 nit (optional): Header comments that describe a whole config and a whole override section break the AGENTS.md comment rule and repeat what the YAML below already says. The comment above minimaxm31flash-fp4-mi355x-atom-agentic-dspark (amd-master.yaml:955-960) summarises the entry. Fix: delete both header blocks and keep only the short line-level explanations, such as the dram-utilization arithmetic and the ATOM_FORCE_ATTN_TRITON reason. The second block is the "One variant per point..." comment at agentic.yaml:57-63.
Why this was flagged
AGENTS.md (Comments, mandatory, every file) says: "No header comments describing a file, topology, config, recipe, override, or section" and "never restate the code". amd-master.yaml:955-960 is a six-line header over the new config key. It restates the image, quantization, TP4, the DSpark draft, which concurrencies use the prefill delayer and LMCache, and the golden AL curve, all of which the entry and recipe already encode. The recipe's agentic.yaml:57-63 block heads the override section and repeats max-num-seqs, cudagraph-capture-sizes, ATOM_PREFILL_DECODE_INTERVAL and LMACHE sizes that every override spells out. These comments drift as soon as a value changes, which is why the rule exists. This affects maintainability only; runtime behaviour does not change.
Verification: AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." In amd-master.yaml the diff adds a six-line header directly above minimaxm31flash-fp4-mi355x-atom-agentic-dspark:, at about lines 955-960. In agentic.yaml a free-standing block describes the whole override section. The comments do not change runtime behavior.
|
LGTM |
f2884f6 to
03dc461
Compare
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42457076638 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42457076638 |
03dc461 to
182ed8f
Compare
…served as shipped / feat(agentx):新增 MiniMax-M3.1-Flash FP4 MI355X ATOM AgentX,Quark MXFP4 按原样服务
182ed8f to
83c3767
Compare
|
/use 42457076638 |
|
@zhuyuhua-v staged run 42457076638. This run remains available across future |
…atom-agentic-1106 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/reuse-sweep-run 42457076638 |
…h the diff - New MiniMax-M3.1-Flash AgentX config on MI355X with ATOM: image rocm/atom-dev:nightly_202611060646 (sha256:857297cc, ATOM 0.1.10.dev36+ge1295074f, AITER b0e8ce03d), serving amd/MiniMax-M3.1-Flash-Quark-MXFP4 as shipped. - Points: TP4 at concurrency 1, 2, 4 and 8 GPU-resident, and at concurrency 16, 32 and 48 with the LMCache 0.5.6.dev139+gf22dec28 CPU tier (dram-utilization 0.58). - Server settings: FP8 KV and index cache, 128-token blocks, prefix caching, gpu-memory-utilization 0.9, max-num-batched-tokens 32768, ATOM_FORCE_ATTN_TRITON=1, max-num-seqs at twice the concurrency with matching decode graphs, the prefill delayer on concurrency 1 to 16, 200 GiB of LMCache per rank, and the bundled seven-token DSpark draft at golden AL 4.04. - Relevant ATOM changes in the image: ROCm/ATOM#2678 and the MiniMax-M3 LMCache offload layout (#2140, #2166); #2686 changes only mono decode, which is not enabled. - Draft data type: the DSpark draft runs as shipped (MXFP8 linears, BF16 heads, FP8 draft KV); no accuracy-sensitive optimization is enabled. - Drop the header comments that restated the config and override section (AGENTS.md comment rule); keep the line-level reasons. Comments only, no setting changes. - 新增 MI355X 上 ATOM 的 MiniMax-M3.1-Flash AgentX 配置:镜像 rocm/atom-dev:nightly_202611060646(sha256:857297cc,ATOM 0.1.10.dev36+ge1295074f,AITER b0e8ce03d),按原样服务 amd/MiniMax-M3.1-Flash-Quark-MXFP4。 - 点位:TP4,并发 1、2、4、8 只用显存;并发 16、32、48 使用 LMCache 0.5.6.dev139+gf22dec28 CPU 层(dram-utilization 0.58)。 - 服务配置:FP8 KV 和 index cache,128-token block,开启 prefix caching,gpu-memory-utilization 0.9,max-num-batched-tokens 32768,ATOM_FORCE_ATTN_TRITON=1,max-num-seqs 为并发数 2 倍并配套 decode graph,并发 1 到 16 开启 prefill delayer,每个 rank 200 GiB LMCache,自带的 7-token DSpark draft,golden AL 4.04。 - 镜像中相关的 ATOM 变更:ROCm/ATOM#2678 以及 MiniMax-M3 LMCache offload layout(#2140、#2166);#2686 只改 mono decode,本配置未启用。 - Draft 数据类型:DSpark draft 按出厂精度运行(MXFP8 线性层、BF16 head、FP8 draft KV);未启用任何精度敏感优化。 - 删除重复描述配置和 override 段落的头部注释(AGENTS.md 注释规则),保留逐行说明。仅改注释,不改设置。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Draft data types: no change. The DSpark draft (
At ATOM Accuracy-sensitive optimizations: none used. Checked against the accuracy-sensitive optimization policy (#4705, still open) for the aggregated TP4 server and its draft worker, including image defaults at ATOM
None of these optimizations is used, so the three-arm qualification suite is not required. The PR and its linked issues contain no model-trainer veto. Accuracy check: GSM8K strict exact match is 0.9666 at C8, 0.9629 at C16 and 0.9644 at C48, each over 1,319 questions. 中文Draft 数据类型:未改变。 DSpark draft(
在 ATOM 精度敏感优化:未使用。 已对照精度敏感优化策略(#4705,尚未合入)检查聚合 TP4 server 及其 draft worker,包括 ATOM
以上优化均未使用,因此不需要三组对照的资格评测。PR 及其关联 issue 中没有模型训练方的否决。精度检查:GSM8K strict exact match 在 C8 为 0.9666,C16 为 0.9629,C48 为 0.9644,均为 1,319 题。 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Draft data types and accuracy-sensitive optimizations: #4860 (comment)
- Recipes:
Signed: seungrokj
|
FYI. amd/MiniMax-M3.1-Flash-Quark-MXFP4 is already used in minimaxm31flash-fp4-mi355x-vllm-agentic-dspark: |
|
|
@zhuyuhua-v is this perf expected and can be merged as it is ? |

Summary
This PR adds
minimaxm31flash-fp4-mi355x-atom-agentic-dspark: MiniMax-M3.1-Flash AgentX on MI355X with ATOM. It follows the M3.1 section of the ATOM agentic recipe (ROCm/ATOM#2747) and needs nothing beyond ATOMmain.rocm/atom-dev:nightly_202611060646@sha256:857297cc…(ATOM0.1.10.dev36+ge1295074f). The atomesh router uses the same digest.amd/MiniMax-M3.1-Flash-Quark-MXFP4, served as shipped. Routed experts are MXFP4; attention, dense MLP and the shared expert are MXFP8 (1x32). The recipe sets:online_quant_config;ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4);AITER_QUICK_REDUCE_QUANTIZATION.max-num-batched-tokens32768,gpu-memory-utilization0.9;max-num-seqs) is twice the concurrency, at least 2;ATOM_PREFILL_DECODE_INTERVAL=20000); C32 and C48 run without it.lmcache_offload, LMCache0.5.6.dev139+gf22dec28, the wheel in the image) at 200 GiB per rank. That is 800 GiB (858.99 GB) per TP4 server, within the 869 GB thatdram-utilization0.58 gives a TP4 server (2,861,022 MiB x 0.58 x 4/8). C1-C8 stay GPU-resident.dspark/draft, seven tokens, at its shipped MXFP8 precision. Throughput runs take the golden AL fromminimax-m3.1-flash__dspark__native__probabilistic-block(4.04 at K7) through the synthetic-acceptance connector; the recipe hard-codes no AL.use_index_cache/index_topk_freq, and ATOM shares top-k only whenuse_index_cacheis set. This matches the SGLang recipe'sSGLANG_MINIMAX_M3_INDEX_TOPK_FREQ=1.Generated matrix: 7 throughput jobs (TP4 C1/2/4/8 GPU-resident, C16/32/48 with LMCache), plus the default per-engine-environment GSM8K at C8, C16 and C48.
Validation. Sweep run 42457076638 on
83c37679epassed all jobs. The recipe and this config key are unchanged since then; the later merge frommainbrings in only other keys.GSM8K strict exact match (1,319 questions): C8 0.9666, C16 0.9629, C48 0.9644.
Throughput (total tok/s/GPU, p90 end-to-end latency):
Throughput runs used forced acceptance at the golden AL 4.04 (logged as
Forced speculative acceptance ON: mean acceptance length 4.0400). The eval runs measured an AL of 3.59, which is reported only.Accuracy-sensitive optimizations: none used. The PR uses none of the optimizations in the accuracy-sensitive optimization policy (#4705, still open), including image defaults at ATOM
e1295074f/ AITERb0e8ce03d:The evidence is in the draft data-type and accuracy-sensitive optimization comment on this PR.
Why C16-C48 add the CPU tier. With FP8 KV and index cache the TP4 GPU pool holds about 7.3M tokens at
gpu-memory-utilization0.9. That figure is derived: 57,011 blocks x 128 tokens, withblock_bytes=3179520, both as recorded in the ATOM recipe from an earlier nightly's server log. An earlier C48 run with these caches was memory-bound: 0.8 utilization and the older command. It had a 19.9% token-weighted prefix hit, 44 s TTFT p50 and about 94 output tok/s/GPU.Validation details
LMCache on MiniMax-M3.1 (ATOM
e1295074f).lmcache_offloadresolves them3layout, which stores every PAGE region the model runner publishes. That covers K, V, scales and the NSA index cache (atom/kv_transfer/offload/hybrid/m3/connector.py), with one shard per TP rank.model_runner.pymerges its regions into the same transfer tensors (merge_pages), so a restored prefix carries the draft's context KV too.PYTHONHASHSEED=0keeps the per-rank cache keys equal.lookup_server_worker_ids,lmcache/v1/config.py).OFFLOAD_MIN_LOAD_TOKENSkeeps ATOM's default, 8192.Draft precision.
dspark/ofamd/MiniMax-M3.1-Flash-Quark-MXFP4(weights unchanged since revision0a00e0e3; later commits touch only the README). ATOM takes the draft as a local directory and does not fetchdspark/itself, so the recipe pointsdraft-modelat/mnt/hf_hub_cache/models--MiniMaxAI--MiniMax-M3.1-Flash/snapshots/2f33a8b9…/dspark, the shared-cache copy that the SGLang M3.1 MI355X runs load (run 41277894342 logsdefaulting --speculative-draft-model-path tothat path). Its three files have the same git blob oids as this checkpoint'sdspark/, so the draft is the shipped one. It is byte-identical todspark/ofMiniMaxAI/MiniMax-M3.1-Flash. The Hugging Face tree API gives both repos the same LFS oid formodel-dspark-00001.safetensors(a27aed923fa17697…, 2,301,867,426 bytes) and the same oids forconfig.jsonand the index.quant_method: mxfp8,weight_block_size: [1, 32]).fc, the Markov head and the confidence head are listed as ignored, so they stay BF16.atom/models/minimax_m3/dspark.py). The recipe sets no online quantization and no draft-related environment.--kv_cache_dtype fp8.Integer all-reduce.
AITER_QUICK_REDUCE_QUANTIZATIONis unset, and nothing in the effective environment enables integer all-reduce:/app/aiter-test,b0e8ce03d),quick_all_reduce.pyreads an unset value asNONEand disables quick reduce.e1295074fsetsINT4itself only forllamamodels in BF16/FP16 (model_runner.py).Image provenance. The digest is the registry
Docker-Content-Digestforrocm/atom-dev:nightly_202611060646. The image labels are:org.opencontainers.image.revision: e1295074fffb4d1a4a99a33998cdf6100ab8c3a4;org.opencontainers.image.version: 0.1.10.dev36+ge1295074f.That revision contains ROCm/ATOM#2678 (MiniMax-M3.1-Flash day-0 support) and #2686.
AI model disclosure
claude-opus-5-5), via Claude Code.claude-opus-5-5), via Claude Code, for seungrokj's review.e1295074fand AITERb0e8ce03d.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
本 PR 新增
minimaxm31flash-fp4-mi355x-atom-agentic-dspark:在 MI355X 上用 ATOM 跑 MiniMax-M3.1-Flash AgentX。配置沿用 ATOM agentic recipe 中的 M3.1 章节(ROCm/ATOM#2747),只依赖 ATOMmain。rocm/atom-dev:nightly_202611060646@sha256:857297cc…(ATOM0.1.10.dev36+ge1295074f),atomesh router 使用同一 digest。amd/MiniMax-M3.1-Flash-Quark-MXFP4,按原样服务。routed experts 为 MXFP4;attention、dense MLP 和 shared expert 为 MXFP8(1x32)。recipe 中:online_quant_config;ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4);AITER_QUICK_REDUCE_QUANTIZATION。max-num-batched-tokens32768,gpu-memory-utilization0.9;max-num-seqs为并发数的 2 倍,最少为 2;max-num-seqs;ATOM_PREFILL_DECODE_INTERVAL=20000),C32 和 C48 不开。lmcache_offload,LMCache0.5.6.dev139+gf22dec28,即镜像内的 wheel),每个 rank 200 GiB。每台 TP4 server 共 800 GiB(858.99 GB),在dram-utilization0.58 给 TP4 server 的 869 GB 额度之内(2,861,022 MiB × 0.58 × 4/8)。C1-C8 仍只用 GPU 显存。dspark/draft,7 个 token,保持其出厂 MXFP8 精度。吞吐测试由 synthetic acceptance connector 从minimax-m3.1-flash__dspark__native__probabilistic-block取 golden AL(K7 为 4.04),recipe 中没有写死 AL。use_index_cache/index_topk_freq,而 ATOM 只有在设置use_index_cache时才共享 top-k。这与 SGLang recipe 中的SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ=1一致。生成的矩阵:7 个吞吐任务(TP4 C1/2/4/8 只用显存,C16/32/48 带 LMCache),外加按引擎环境默认选出的 GSM8K:C8、C16 和 C48。
验证。 Sweep run 42457076638 在
83c37679e上全部通过。此后 recipe 和本 config key 都没有变化;之后从main合入的提交只涉及其他 key。GSM8K strict exact match(1,319 题):C8 0.9666,C16 0.9629,C48 0.9644。
吞吐(总 tok/s/GPU,p90 端到端延迟):
吞吐测试按 golden AL 4.04 强制接受(日志为
Forced speculative acceptance ON: mean acceptance length 4.0400)。eval 测得的 AL 为 3.59,仅作记录。精度敏感优化:未使用。 本 PR 未使用精度敏感优化策略(#4705,尚未合入)所列的任何优化,ATOM
e1295074f/ AITERb0e8ce03d的镜像默认值也已核对:证据见本 PR 上关于 draft 数据类型和精度敏感优化的评论。
C16-C48 为何增加 CPU 层。 FP8 KV 和 index cache 在
gpu-memory-utilization0.9 下,TP4 显存 pool 约可容纳 730 万 token。该数字为推算:57,011 个 block × 128 token,block_bytes=3179520;两者均为 ATOM recipe 中记录的、较早 nightly 的 server 日志数值。此前一次使用这两种 cache 的 C48 测试(利用率 0.8、旧命令)受显存限制:按 token 加权的 prefix 命中率为 19.9%,TTFT p50 为 44 s,output 约 94 tok/s/GPU。MiniMax-M3.1 上的 LMCache(ATOM
e1295074f):lmcache_offload选择m3layout,存储 model runner 发布的全部 PAGE 区域,包括 K、V、scale 和 NSA index cache(atom/kv_transfer/offload/hybrid/m3/connector.py),每个 TP rank 各存一份分片。model_runner.py通过merge_pages把它的区域并入同一份 transfer tensors,因此恢复的 prefix 也包含 draft 的 context KV。PYTHONHASHSEED=0保证各 rank 的 cache key 一致。lookup_server_worker_ids,lmcache/v1/config.py)。OFFLOAD_MIN_LOAD_TOKENS保持 ATOM 默认值 8192。Draft 精度:
amd/MiniMax-M3.1-Flash-Quark-MXFP4的dspark/(权重自 revision0a00e0e3起未变,之后的提交只改 README)。ATOM 要求 draft 为本地目录,且不会自行下载dspark/,因此 recipe 将draft-model指向/mnt/hf_hub_cache/models--MiniMaxAI--MiniMax-M3.1-Flash/snapshots/2f33a8b9…/dspark,即 SGLang M3.1 MI355X 运行所加载的共享缓存副本(run 41277894342 日志中defaulting --speculative-draft-model-path to该路径)。其三个文件的 git blob oid 与本 checkpoint 的dspark/完全相同,draft 仍为出厂版本。它与MiniMaxAI/MiniMax-M3.1-Flash的dspark/逐字节相同:Hugging Face tree API 显示两边model-dspark-00001.safetensors的 LFS oid 都是a27aed923fa17697…,config.json和 index 的 oid 也相同。quant_method: mxfp8,weight_block_size: [1, 32]);fc、Markov head 和 confidence head 列在 ignored 中,保持 BF16。--kv_cache_dtype fp8。整数 all-reduce: 未设置
AITER_QUICK_REDUCE_QUANTIZATION,实际环境中也没有任何地方开启整数 all-reduce:/app/aiter-test,b0e8ce03d)的quick_all_reduce.py将未设置视为NONE,quick reduce 关闭;e1295074f仅对 BF16/FP16 的llama模型自动设为INT4(model_runner.py)。镜像来源: digest 为 registry 对
rocm/atom-dev:nightly_202611060646返回的Docker-Content-Digest。镜像 label 为:org.opencontainers.image.revision: e1295074f…;org.opencontainers.image.version: 0.1.10.dev36+ge1295074f。该 revision 包含 ROCm/ATOM#2678(MiniMax-M3.1-Flash day-0 支持)和 #2686。
AI 使用说明:
claude-opus-5-5,通过 Claude Code)编写 recipe、master config 和 changelog 条目,核对上述镜像、checkpoint 和 policy 要点,并起草本说明;由 zhuyuhua-v 审阅并提交。claude-opus-5-5,通过 Claude Code)在 seungrokj 审阅时按 diff 重写 perf-changelog 条目,补充验证和精度敏感优化部分,并对照 ATOMe1295074f和 AITERb0e8ce03d核对 draft 数据类型和精度敏感优化。