Skip to content

feat(agentx): MiniMax-M3.1-Flash FP4 MI355X ATOM AgentX, Quark MXFP4 served as shipped / feat(agentx):新增 MiniMax-M3.1-Flash FP4 MI355X ATOM AgentX,Quark MXFP4 按原样服务 - #4860

Open
zhuyuhua-v wants to merge 3 commits into
mainfrom
amd/m31-flash-mi355x-atom-agentic-1106
Open

zhuyuhua-v wants to merge 3 commits into
mainfrom
amd/m31-flash-mi355x-atom-agentic-1106

Conversation

@zhuyuhua-v

@zhuyuhua-v zhuyuhua-v commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR adds minimaxm31flash-fp4-mi355x-atom-agentic-dspark: MiniMax-M3.1-Flash AgentX on MI355X with ATOM. It follows the M3.1 section of the ATOM agentic recipe (ROCm/ATOM#2747) and needs nothing beyond ATOM main.

  • Image: rocm/atom-dev:nightly_202611060646@sha256:857297cc… (ATOM 0.1.10.dev36+ge1295074f). The atomesh router uses the same digest.
  • Checkpoint: amd/MiniMax-M3.1-Flash-Quark-MXFP4, served as shipped. Routed experts are MXFP4; attention, dense MLP and the shared expert are MXFP8 (1x32). The recipe sets:
    • no online_quant_config;
    • no shared-expert fusion (ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4);
    • no AITER_QUICK_REDUCE_QUANTIZATION.
  • Serving:
    • TP4, FP8 KV and FP8 index cache, 128-token blocks, prefix caching, max-num-batched-tokens 32768, gpu-memory-utilization 0.9;
    • admission (max-num-seqs) is twice the concurrency, at least 2;
    • decode graphs cover every batch size up to 8, then steps of 2 / 4 / 8 / 16 up to the admission;
    • C1-C16 run the TP-local prefill delayer (ATOM_PREFILL_DECODE_INTERVAL=20000); C32 and C48 run without it.
  • KV offload: C16, C32 and C48 add ATOM's in-engine LMCache CPU tier (lmcache_offload, LMCache 0.5.6.dev139+gf22dec28, the wheel in the image) at 200 GiB per rank. That is 800 GiB (858.99 GB) per TP4 server, within the 869 GB that dram-utilization 0.58 gives a TP4 server (2,861,022 MiB x 0.58 x 4/8). C1-C8 stay GPU-resident.
  • Speculative decoding: the checkpoint's bundled dspark/ draft, seven tokens, at its shipped MXFP8 precision. Throughput runs take the golden AL from minimax-m3.1-flash__dspark__native__probabilistic-block (4.04 at K7) through the synthetic-acceptance connector; the recipe hard-codes no AL.
  • Indexer: every sparse layer selects its own top-k. The checkpoint config has no use_index_cache / index_topk_freq, and ATOM shares top-k only when use_index_cache is set. This matches the SGLang recipe's SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ=1.
  • Mono decode is not used. ATOM's MiniMax-M3 mono kernels need per-token FP8 attention and a fused shared expert, so every point runs the production decode path.

Generated matrix: 7 throughput jobs (TP4 C1/2/4/8 GPU-resident, C16/32/48 with LMCache), plus the default per-engine-environment GSM8K at C8, C16 and C48.

Validation. Sweep run 42457076638 on 83c37679e passed all jobs. The recipe and this config key are unchanged since then; the later merge from main brings in only other keys.

  • GSM8K strict exact match (1,319 questions): C8 0.9666, C16 0.9629, C48 0.9644.

  • Throughput (total tok/s/GPU, p90 end-to-end latency):

    Conc 1 2 4 8 16 32 48
    tok/s/GPU 4,916 5,779 8,233 10,650 19,498 47,248 39,253
    p90 E2EL (s) 19.1 13.8 13.4 22.6 31.7 22.8 71.1
  • Throughput runs used forced acceptance at the golden AL 4.04 (logged as Forced speculative acceptance ON: mean acceptance length 4.0400). The eval runs measured an AL of 3.59, which is reported only.

Accuracy-sensitive optimizations: none used. The PR uses none of the optimizations in the accuracy-sensitive optimization policy (#4705, still open), including image defaults at ATOM e1295074f / AITER b0e8ce03d:

  • quick-reduce quantization is off;
  • there is no online quantization;
  • shared experts stay MXFP8, as shipped, and are not fused;
  • indexers are not shared;
  • the KV cache is FP8;
  • the model has no SSM state.

The evidence is in the draft data-type and accuracy-sensitive optimization comment on this PR.

Why C16-C48 add the CPU tier. With FP8 KV and index cache the TP4 GPU pool holds about 7.3M tokens at gpu-memory-utilization 0.9. That figure is derived: 57,011 blocks x 128 tokens, with block_bytes=3179520, both as recorded in the ATOM recipe from an earlier nightly's server log. An earlier C48 run with these caches was memory-bound: 0.8 utilization and the older command. It had a 19.9% token-weighted prefix hit, 44 s TTFT p50 and about 94 output tok/s/GPU.

Validation details

LMCache on MiniMax-M3.1 (ATOM e1295074f).

  • lmcache_offload resolves the m3 layout, which stores every PAGE region the model runner publishes. That covers K, V, scales and the NSA index cache (atom/kv_transfer/offload/hybrid/m3/connector.py), with one shard per TP rank.
  • The DSpark draft's KV pool rides the target's block ids. model_runner.py merges its regions into the same transfer tensors (merge_pages), so a restored prefix carries the draft's context KV too.
  • PYTHONHASHSEED=0 keeps the per-rank cache keys equal.
  • LMCache starts its lookup server on every rank by default for non-MLA models (lookup_server_worker_ids, lmcache/v1/config.py).
  • OFFLOAD_MIN_LOAD_TOKENS keeps ATOM's default, 8192.

Draft precision.

  • Draft: dspark/ of amd/MiniMax-M3.1-Flash-Quark-MXFP4 (weights unchanged since revision 0a00e0e3; later commits touch only the README). ATOM takes the draft as a local directory and does not fetch dspark/ itself, so the recipe points draft-model at /mnt/hf_hub_cache/models--MiniMaxAI--MiniMax-M3.1-Flash/snapshots/2f33a8b9…/dspark, the shared-cache copy that the SGLang M3.1 MI355X runs load (run 41277894342 logs defaulting --speculative-draft-model-path to that path). Its three files have the same git blob oids as this checkpoint's dspark/, so the draft is the shipped one. It is byte-identical to dspark/ of MiniMaxAI/MiniMax-M3.1-Flash. The Hugging Face tree API gives both repos the same LFS oid for model-dspark-00001.safetensors (a27aed923fa17697…, 2,301,867,426 bytes) and the same oids for config.json and the index.
  • Shipped precision: MXFP8 linears (quant_method: mxfp8, weight_block_size: [1, 32]). fc, the Markov head and the confidence head are listed as ignored, so they stay BF16.
  • Pinned image's default: ATOM builds the draft's quantization from the draft's own config, independent of the target's. It loads the MXFP8 linears natively (atom/models/minimax_m3/dspark.py). The recipe sets no online quantization and no draft-related environment.
  • Draft KV: FP8, inherited from the target's --kv_cache_dtype fp8.
  • Effective serving precision: MXFP8, the shipped precision.

Integer all-reduce. AITER_QUICK_REDUCE_QUANTIZATION is unset, and nothing in the effective environment enables integer all-reduce:

  • In the image's AITER (/app/aiter-test, b0e8ce03d), quick_all_reduce.py reads an unset value as NONE and disables quick reduce.
  • ATOM e1295074f sets INT4 itself only for llama models in BF16/FP16 (model_runner.py).

Image provenance. The digest is the registry Docker-Content-Digest for rocm/atom-dev:nightly_202611060646. The image labels are:

  • org.opencontainers.image.revision: e1295074fffb4d1a4a99a33998cdf6100ab8c3a4;
  • org.opencontainers.image.version: 0.1.10.dev36+ge1295074f.

That revision contains ROCm/ATOM#2678 (MiniMax-M3.1-Flash day-0 support) and #2686.

AI model disclosure
  • Model/version: Claude Opus 5.5 (claude-opus-5-5), via Claude Code.
  • Role: wrote the recipe, master-config entry and changelog entry (LMCache tier included), checked the image, checkpoint, LMCache and policy points above, and drafted this description; zhuyuhua-v reviewed and submitted it.
  • Model/version: Claude Opus 5.5 (claude-opus-5-5), via Claude Code, for seungrokj's review.
  • Role: rewrote the perf-changelog block to match the diff and added the validation and accuracy-sensitive optimization sections. It also checked the draft data types and the accuracy-sensitive optimizations against ATOM e1295074f and AITER b0e8ce03d.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

本 PR 新增 minimaxm31flash-fp4-mi355x-atom-agentic-dspark:在 MI355X 上用 ATOM 跑 MiniMax-M3.1-Flash AgentX。配置沿用 ATOM agentic recipe 中的 M3.1 章节(ROCm/ATOM#2747),只依赖 ATOM main。

  • 镜像: rocm/atom-dev:nightly_202611060646@sha256:857297cc…(ATOM 0.1.10.dev36+ge1295074f),atomesh router 使用同一 digest。
  • Checkpoint: amd/MiniMax-M3.1-Flash-Quark-MXFP4,按原样服务。routed experts 为 MXFP4;attention、dense MLP 和 shared expert 为 MXFP8(1x32)。recipe 中:
    • 不设 online_quant_config;
    • 不做 shared expert 融合(不设 ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4);
    • 不设 AITER_QUICK_REDUCE_QUANTIZATION。
  • 服务配置:
    • TP4,FP8 KV 和 FP8 index cache,128-token block,开启 prefix caching,max-num-batched-tokens 32768,gpu-memory-utilization 0.9;
    • max-num-seqs 为并发数的 2 倍,最少为 2;
    • decode graph 覆盖 1-8 的每个 batch size,此后按 2 / 4 / 8 / 16 步长递增到 max-num-seqs;
    • C1-C16 开启 TP 内 prefill delayer(ATOM_PREFILL_DECODE_INTERVAL=20000),C32 和 C48 不开。
  • KV offload: C16、C32 和 C48 增加 ATOM 进程内 LMCache CPU 层(lmcache_offload,LMCache 0.5.6.dev139+gf22dec28,即镜像内的 wheel),每个 rank 200 GiB。每台 TP4 server 共 800 GiB(858.99 GB),在 dram-utilization 0.58 给 TP4 server 的 869 GB 额度之内(2,861,022 MiB × 0.58 × 4/8)。C1-C8 仍只用 GPU 显存。
  • 投机解码: 使用 checkpoint 自带的 dspark/ draft,7 个 token,保持其出厂 MXFP8 精度。吞吐测试由 synthetic acceptance connector 从 minimax-m3.1-flash__dspark__native__probabilistic-block 取 golden AL(K7 为 4.04),recipe 中没有写死 AL。
  • Indexer: 每个 sparse 层都自己计算 top-k。checkpoint config 中没有 use_index_cache / index_topk_freq,而 ATOM 只有在设置 use_index_cache 时才共享 top-k。这与 SGLang recipe 中的 SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ=1 一致。
  • Mono decode 不启用。ATOM 的 MiniMax-M3 mono kernel 需要 per-token FP8 attention 和融合后的 shared expert,因此所有点位都走普通 decode 路径。

生成的矩阵:7 个吞吐任务(TP4 C1/2/4/8 只用显存,C16/32/48 带 LMCache),外加按引擎环境默认选出的 GSM8K:C8、C16 和 C48。

验证。 Sweep run 42457076638 在 83c37679e 上全部通过。此后 recipe 和本 config key 都没有变化;之后从 main 合入的提交只涉及其他 key。

  • GSM8K strict exact match(1,319 题):C8 0.9666,C16 0.9629,C48 0.9644。

  • 吞吐(总 tok/s/GPU,p90 端到端延迟):

    并发 1 2 4 8 16 32 48
    tok/s/GPU 4,916 5,779 8,233 10,650 19,498 47,248 39,253
    p90 E2EL (s) 19.1 13.8 13.4 22.6 31.7 22.8 71.1
  • 吞吐测试按 golden AL 4.04 强制接受(日志为 Forced speculative acceptance ON: mean acceptance length 4.0400)。eval 测得的 AL 为 3.59,仅作记录。

精度敏感优化:未使用。 本 PR 未使用精度敏感优化策略(#4705,尚未合入)所列的任何优化,ATOM e1295074f / AITER b0e8ce03d 的镜像默认值也已核对:

  • quick reduce 量化关闭;
  • 没有在线量化;
  • shared expert 保持出厂的 MXFP8,且未融合;
  • 不共享 indexer;
  • KV cache 为 FP8;
  • 模型没有 SSM 状态。

证据见本 PR 上关于 draft 数据类型和精度敏感优化的评论。

C16-C48 为何增加 CPU 层。 FP8 KV 和 index cache 在 gpu-memory-utilization 0.9 下,TP4 显存 pool 约可容纳 730 万 token。该数字为推算:57,011 个 block × 128 token,block_bytes=3179520;两者均为 ATOM recipe 中记录的、较早 nightly 的 server 日志数值。此前一次使用这两种 cache 的 C48 测试(利用率 0.8、旧命令)受显存限制:按 token 加权的 prefix 命中率为 19.9%,TTFT p50 为 44 s,output 约 94 tok/s/GPU。

MiniMax-M3.1 上的 LMCache(ATOM e1295074f):

  • lmcache_offload 选择 m3 layout,存储 model runner 发布的全部 PAGE 区域,包括 K、V、scale 和 NSA index cache(atom/kv_transfer/offload/hybrid/m3/connector.py),每个 TP rank 各存一份分片。
  • DSpark draft 的 KV pool 复用 target 的 block id,model_runner.py 通过 merge_pages 把它的区域并入同一份 transfer tensors,因此恢复的 prefix 也包含 draft 的 context KV。
  • PYTHONHASHSEED=0 保证各 rank 的 cache key 一致。
  • 对非 MLA 模型,LMCache 默认在每个 rank 上启动 lookup server(lookup_server_worker_ids,lmcache/v1/config.py)。
  • OFFLOAD_MIN_LOAD_TOKENS 保持 ATOM 默认值 8192。

Draft 精度:

  • Draft: amd/MiniMax-M3.1-Flash-Quark-MXFP4 的 dspark/(权重自 revision 0a00e0e3 起未变,之后的提交只改 README)。ATOM 要求 draft 为本地目录,且不会自行下载 dspark/,因此 recipe 将 draft-model 指向 /mnt/hf_hub_cache/models--MiniMaxAI--MiniMax-M3.1-Flash/snapshots/2f33a8b9…/dspark,即 SGLang M3.1 MI355X 运行所加载的共享缓存副本(run 41277894342 日志中 defaulting --speculative-draft-model-path to 该路径)。其三个文件的 git blob oid 与本 checkpoint 的 dspark/ 完全相同,draft 仍为出厂版本。它与 MiniMaxAI/MiniMax-M3.1-Flash 的 dspark/ 逐字节相同:Hugging Face tree API 显示两边 model-dspark-00001.safetensors 的 LFS oid 都是 a27aed923fa17697…,config.json 和 index 的 oid 也相同。
  • 出厂精度: MXFP8 线性层(quant_method: mxfp8,weight_block_size: [1, 32]);fc、Markov head 和 confidence head 列在 ignored 中,保持 BF16。
  • 镜像默认处理: ATOM 按 draft 自己的 config 构建其量化,不受 target 影响,直接以原生 MXFP8 加载这些线性层。recipe 中没有在线量化,也没有任何与 draft 相关的环境变量。
  • Draft KV: FP8,沿用 target 的 --kv_cache_dtype fp8。
  • 实际服务精度: MXFP8,与出厂一致。

整数 all-reduce: 未设置 AITER_QUICK_REDUCE_QUANTIZATION,实际环境中也没有任何地方开启整数 all-reduce:

  • 镜像内 AITER(/app/aiter-test,b0e8ce03d)的 quick_all_reduce.py 将未设置视为 NONE,quick reduce 关闭;
  • ATOM e1295074f 仅对 BF16/FP16 的 llama 模型自动设为 INT4(model_runner.py)。

镜像来源: digest 为 registry 对 rocm/atom-dev:nightly_202611060646 返回的 Docker-Content-Digest。镜像 label 为:

  • org.opencontainers.image.revision: e1295074f…;
  • org.opencontainers.image.version: 0.1.10.dev36+ge1295074f。

该 revision 包含 ROCm/ATOM#2678(MiniMax-M3.1-Flash day-0 支持)和 #2686。

AI 使用说明:

  • Claude Opus 5.5(claude-opus-5-5,通过 Claude Code)编写 recipe、master config 和 changelog 条目,核对上述镜像、checkpoint 和 policy 要点,并起草本说明;由 zhuyuhua-v 审阅并提交。
  • Claude Opus 5.5(claude-opus-5-5,通过 Claude Code)在 seungrokj 审阅时按 diff 重写 perf-changelog 条目,补充验证和精度敏感优化部分,并对照 ATOM e1295074f 和 AITER b0e8ce03d 核对 draft 数据类型和精度敏感优化。

@zhuyuhua-v
zhuyuhua-v force-pushed the amd/m31-flash-mi355x-atom-agentic-1106 branch 2 times, most recently from d311cab to f2884f6 Compare November 6, 2026 07:55
@zhuyuhua-v
zhuyuhua-v marked this pull request as ready for review November 6, 2026 07:57
@zhuyuhua-v
zhuyuhua-v requested review from a team and a balanced review from Copilot November 6, 2026 07:57
@zhuyuhua-v zhuyuhua-v added AMD full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures agentx AgentX benchmarks, recipes, and infrastructure labels Nov 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two configuration comments redundantly restate settings and need removal or trimming to comply with the repository’s comment requirements.

2 open issues
What changed in this PR

Adds MiniMax-M3.1-Flash FP4 AgentX benchmarking on MI355X with ATOM, serving the Quark checkpoint and bundled DSpark draft at their shipped precisions.

Changes:

  • Registers seven TP4 concurrency points, with LMCache CPU offloading at C16–C48.
  • Adds a pinned-image recipe with FP8 caches, concurrency-specific decode graphs, and prefill delays at C1–C16.
  • Appends a performance-changelog entry for sweep collection.
File Description
inferencex-e2e/​perf-changelog.yaml Records the new benchmark configuration and tuning.
inferencex-e2e/​configs/​amd-master.yaml Registers the model, image, runner, and concurrency tiers.
inferencex-e2e/​benchmarks/​single_node/​srt-slurm-recipes/​minimaxm31flash/​atom/​mi355x-fp4-mtp/​agentic.yaml Defines ATOM serving, DSpark speculation, and LMCache variants.

Review effort: Balanced

Comment on lines +57 to +63
# One variant per point. Admission is 2x CONC (at least 2); decode graphs cover
# every batch size up to 8, then steps of 2 / 4 / 8 / 16 up to the admission.
# C1-C16 run the TP-local prefill delayer: after a prefill, no new prefill is
# admitted for 20000 scheduler passes while requests decode. C16-C48 add the
# LMCache CPU tier at 200 GiB per rank (800 GiB, 859 GB per server), since the
# FP8 KV and index pool alone (about 7.3M tokens) is smaller than the AgentX
# working set at C48.
Comment thread inferencex-e2e/configs/amd-master.yaml Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

# K7) from infx/srt_slurm/synthetic_acceptance.py; evals keep real
# acceptance.
method: dspark
draft-model: /model/dspark

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Results from this config will record the draft model as the container path /model/dspark, not native like every other bundled-draft config. The cause is the explicit draft-model: /model/dspark at agentic.yaml:42. spec_parameters copies it as the draft model, and speculation_facts only rewrites HF cache paths or an empty value to native. Fix: drop draft-model if ATOM finds the checkpoint's dspark/ itself, as the dsv41flash ATOM recipe does. Otherwise make speculation_facts report native for a draft inside the served checkpoint.

Why this was flagged

For ATOM, spec_parameters (infx/srt_slurm/synthetic_acceptance.py:82) sets model to args.get("draft-model", ""), which is /model/dspark here. speculation_facts (infx/srt_slurm/spec_identity.py:48-54) returns native only when that string is empty. A path matching models--<org>--<name> becomes a repo name. /model/dspark matches neither, so it is passed through unchanged. bind_speculation (infx/srt_slurm/workload.py:479-481) writes it to SPEC_DRAFT_MODEL. speculation_fields (infx/results/metadata.py:45-48) stores that as spec_draft_model in every throughput and eval result for minimaxm31flash-fp4-mi355x-atom-agentic-dspark. The draft is the checkpoint's own bundled dspark/, the same draft the SGLang and vLLM M3.1-Flash configs record as native. No check rejects the path, because compare does not look at the draft field.

Verification: nit. The recipe sets draft-model: /model/dspark (agentic.yaml:42). speculation_facts (infx/srt_slurm/spec_identity.py:48-54) rewrites only HF cache paths (models--<org>--<name>), and returns native only when the value is empty (model or "native"). /model/dspark matches neither, so it passes through unchanged. Curve selection is not affected.

Comment thread inferencex-e2e/configs/amd-master.yaml Outdated
Comment on lines +955 to +960
# MiniMax-M3.1-Flash AgentX on MI355X with ATOM: AMD's Quark MXFP4 export served as
# shipped (no online quantization, no shared-expert fusion, no quick-reduce
# quantization) at TP4 with the bundled seven-token DSpark draft, FP8 KV and FP8
# index cache. C1-C16 run the prefill delayer; C16-C48 add the LMCache CPU tier.
# Throughput runs use the minimax-m3.1-flash__dspark__native__probabilistic-block
# golden AL.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): Header comments that describe a whole config and a whole override section break the AGENTS.md comment rule and repeat what the YAML below already says. The comment above minimaxm31flash-fp4-mi355x-atom-agentic-dspark (amd-master.yaml:955-960) summarises the entry. Fix: delete both header blocks and keep only the short line-level explanations, such as the dram-utilization arithmetic and the ATOM_FORCE_ATTN_TRITON reason. The second block is the "One variant per point..." comment at agentic.yaml:57-63.

Why this was flagged

AGENTS.md (Comments, mandatory, every file) says: "No header comments describing a file, topology, config, recipe, override, or section" and "never restate the code". amd-master.yaml:955-960 is a six-line header over the new config key. It restates the image, quantization, TP4, the DSpark draft, which concurrencies use the prefill delayer and LMCache, and the golden AL curve, all of which the entry and recipe already encode. The recipe's agentic.yaml:57-63 block heads the override section and repeats max-num-seqs, cudagraph-capture-sizes, ATOM_PREFILL_DECODE_INTERVAL and LMACHE sizes that every override spells out. These comments drift as soon as a value changes, which is why the rule exists. This affects maintainability only; runtime behaviour does not change.

Verification: AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." In amd-master.yaml the diff adds a six-line header directly above minimaxm31flash-fp4-mi355x-atom-agentic-dspark:, at about lines 955-960. In agentic.yaml a free-standing block describes the whole override section. The comments do not change runtime behavior.

@zejunchen-zejun

Copy link
Copy Markdown
Collaborator

LGTM

@zhuyuhua-v
zhuyuhua-v force-pushed the amd/m31-flash-mi355x-atom-agentic-1106 branch from f2884f6 to 03dc461 Compare November 8, 2026 04:25
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

@zhuyuhua-v
zhuyuhua-v force-pushed the amd/m31-flash-mi355x-atom-agentic-1106 branch from 03dc461 to 182ed8f Compare November 8, 2026 05:58
…served as shipped / feat(agentx):新增 MiniMax-M3.1-Flash FP4 MI355X ATOM AgentX,Quark MXFP4 按原样服务
@zhuyuhua-v
zhuyuhua-v force-pushed the amd/m31-flash-mi355x-atom-agentic-1106 branch from 182ed8f to 83c3767 Compare November 8, 2026 07:01
@zhuyuhua-v

Copy link
Copy Markdown
Collaborator Author

/use 42457076638

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

@zhuyuhua-v staged run 42457076638.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…atom-agentic-1106

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@seungrokj

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 42457076638

…h the diff

- New MiniMax-M3.1-Flash AgentX config on MI355X with ATOM: image rocm/atom-dev:nightly_202611060646 (sha256:857297cc, ATOM 0.1.10.dev36+ge1295074f, AITER b0e8ce03d), serving amd/MiniMax-M3.1-Flash-Quark-MXFP4 as shipped.
- Points: TP4 at concurrency 1, 2, 4 and 8 GPU-resident, and at concurrency 16, 32 and 48 with the LMCache 0.5.6.dev139+gf22dec28 CPU tier (dram-utilization 0.58).
- Server settings: FP8 KV and index cache, 128-token blocks, prefix caching, gpu-memory-utilization 0.9, max-num-batched-tokens 32768, ATOM_FORCE_ATTN_TRITON=1, max-num-seqs at twice the concurrency with matching decode graphs, the prefill delayer on concurrency 1 to 16, 200 GiB of LMCache per rank, and the bundled seven-token DSpark draft at golden AL 4.04.
- Relevant ATOM changes in the image: ROCm/ATOM#2678 and the MiniMax-M3 LMCache offload layout (#2140, #2166); #2686 changes only mono decode, which is not enabled.
- Draft data type: the DSpark draft runs as shipped (MXFP8 linears, BF16 heads, FP8 draft KV); no accuracy-sensitive optimization is enabled.
- Drop the header comments that restated the config and override section (AGENTS.md comment rule); keep the line-level reasons. Comments only, no setting changes.

- 新增 MI355X 上 ATOM 的 MiniMax-M3.1-Flash AgentX 配置:镜像 rocm/atom-dev:nightly_202611060646(sha256:857297cc,ATOM 0.1.10.dev36+ge1295074f,AITER b0e8ce03d),按原样服务 amd/MiniMax-M3.1-Flash-Quark-MXFP4。
- 点位:TP4,并发 1、2、4、8 只用显存;并发 16、32、48 使用 LMCache 0.5.6.dev139+gf22dec28 CPU 层(dram-utilization 0.58)。
- 服务配置:FP8 KV 和 index cache,128-token block,开启 prefix caching,gpu-memory-utilization 0.9,max-num-batched-tokens 32768,ATOM_FORCE_ATTN_TRITON=1,max-num-seqs 为并发数 2 倍并配套 decode graph,并发 1 到 16 开启 prefill delayer,每个 rank 200 GiB LMCache,自带的 7-token DSpark draft,golden AL 4.04。
- 镜像中相关的 ATOM 变更:ROCm/ATOM#2678 以及 MiniMax-M3 LMCache offload layout(#2140、#2166);#2686 只改 mono decode,本配置未启用。
- Draft 数据类型:DSpark draft 按出厂精度运行(MXFP8 线性层、BF16 head、FP8 draft KV);未启用任何精度敏感优化。
- 删除重复描述配置和 override 段落的头部注释(AGENTS.md 注释规则),保留逐行说明。仅改注释,不改设置。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj

Copy link
Copy Markdown
Collaborator

Draft data types: no change. The DSpark draft (dspark/ in amd/MiniMax-M3.1-Flash-Quark-MXFP4) has the same blobs as dspark/ in MiniMaxAI/MiniMax-M3.1-Flash (snapshot 2f33a8b9, the path the recipe loads). Read from its safetensors header and config.json (quant_method: mxfp8, weight_block_size: [1, 32]), it is stored as:

Part Data type
q/k/v/o projections and dense MLP of the 5 sliding-window layers MXFP8: FP8 E4M3 weights with 1×32 UE8M0 scales
fc, Markov head (w1/w2), confidence head, norms BF16 (listed as ignored layers)
Embedding and LM head not in the draft; shared with the target
Draft KV cache FP8, inherited from the target's --kv_cache_dtype fp8

At ATOM e1295074f (the image's revision), the draft's quantization comes from its own config.json, not from the target's (_draft_quant_config, called here). It accepts only MXFP8 [1, 32] or BF16, and fc is built with quant_config=None. Draft attention takes the target's kv_cache_dtype. The recipe sets no online_quant_config (the server log shows online_quant_config=None) and no draft-related environment, so the draft runs as shipped: MXFP8 linears with BF16 heads and an FP8 KV cache. This is a new config, so there is no base to compare against.

Accuracy-sensitive optimizations: none used. Checked against the accuracy-sensitive optimization policy (#4705, still open) for the aggregated TP4 server and its draft worker, including image defaults at ATOM e1295074f and AITER b0e8ce03d (both confirmed by the runtime fingerprint in run 42457076638):

Optimization Effective setting (new config) Evidence
Reduced-precision all-reduce / reduce-scatter Off. AITER_QUICK_REDUCE_QUANTIZATION is unset; AITER reads it as NONE and disables quick reduce. ATOM forces INT4 only for llama in BF16/FP16. The Gluon fused all-reduce runs only on gfx1250, not on MI355X (gfx950). AITER quick_all_reduce.py#L374, #L384; ATOM model_runner.py#L804-L808; AITER communicator_cuda.py#L142-L145
Online linear-layer quantization Off. There is no online_quant_config or --quantization override; the Quark MXFP4 checkpoint is served in its stored format (online_quant_config=None in the server's engine args). recipe agentic.yaml; server log
Shared-expert quantization Off. The shared expert stays MXFP8 as shipped, the same as MiniMaxAI's release. Fusion into the MXFP4 experts (ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4) defaults to 0 and is not set. envs.py#L1034-L1036
Non-native cross-indexer sharing Off. Every sparse layer computes its own top-k: the checkpoint config has no use_index_cache / index_topk_freq, ATOM shares top-k only when use_index_cache is set, and hf_overrides is None. model.py#L128-L129
Non-native FP4 KV cache Off. The KV and index caches are FP8 (kv_cache_dtype: fp8, index-cache-dtype: fp8). recipe agentic.yaml
SSM/KDA/Mamba state cache below FP32 Not applicable. MiniMax-M3.1-Flash has no SSM, KDA or Mamba state. model config

None of these optimizations is used, so the three-arm qualification suite is not required. The PR and its linked issues contain no model-trainer veto. Accuracy check: GSM8K strict exact match is 0.9666 at C8, 0.9629 at C16 and 0.9644 at C48, each over 1,319 questions.

中文

Draft 数据类型:未改变。 DSpark draft(amd/MiniMax-M3.1-Flash-Quark-MXFP4 中的 dspark/)与 MiniMaxAI/MiniMax-M3.1-Flash 的 dspark/ blob 完全相同(snapshot 2f33a8b9,即 recipe 加载的路径)。根据其 safetensors header 和 config.json(quant_method: mxfp8,weight_block_size: [1, 32]),存储格式如下:

部分 数据类型
5 个 sliding-window 层的 q/k/v/o 投影和 dense MLP MXFP8:FP8 E4M3 权重,1×32 UE8M0 scale
fc、Markov head(w1/w2)、confidence head、norm BF16(列为 ignored 层)
Embedding 和 LM head 不在 draft 中,与 target 共享
Draft KV cache FP8,沿用 target 的 --kv_cache_dtype fp8

在 ATOM e1295074f(镜像对应的 revision)中,draft 的量化配置来自它自己的 config.json,而不是 target 的(_draft_quant_config,调用处)。它只接受 MXFP8 [1, 32] 或 BF16,fc 以 quant_config=None 构建。Draft attention 使用 target 的 kv_cache_dtype。recipe 未设置 online_quant_config(server 日志为 online_quant_config=None),也没有任何与 draft 相关的环境变量,因此 draft 按出厂精度运行:MXFP8 线性层、BF16 head、FP8 KV cache。这是新增配置,没有可对比的 base。

精度敏感优化:未使用。 已对照精度敏感优化策略(#4705,尚未合入)检查聚合 TP4 server 及其 draft worker,包括 ATOM e1295074f 和 AITER b0e8ce03d 的镜像默认值(两者均由 run 42457076638 的运行时指纹确认):

优化 实际设置(新增配置) 证据
低精度 all-reduce / reduce-scatter 关闭。AITER_QUICK_REDUCE_QUANTIZATION 未设置;AITER 将其视为 NONE,quick reduce 关闭。ATOM 只对 BF16/FP16 的 llama 强制设为 INT4。Gluon fused all-reduce 只在 gfx1250 上启用,MI355X(gfx950)上不启用。 AITER quick_all_reduce.py#L374、#L384;ATOM model_runner.py#L804-L808;AITER communicator_cuda.py#L142-L145
在线线性层量化 关闭。没有 online_quant_config 或 --quantization 覆盖;Quark MXFP4 checkpoint 按存储格式服务(server engine 参数中为 online_quant_config=None)。 recipe agentic.yaml;server 日志
Shared expert 量化 关闭。shared expert 保持出厂的 MXFP8,与 MiniMaxAI 官方发布一致。融合到 MXFP4 expert 的开关(ATOM_FUSE_MXFP8_SHARED_INTO_MXFP4)默认为 0,且未设置。 envs.py#L1034-L1036
非原生跨 indexer 共享 关闭。每个 sparse 层各自计算 top-k:checkpoint config 中没有 use_index_cache / index_topk_freq,ATOM 只有在设置 use_index_cache 时才共享 top-k,hf_overrides 为 None。 model.py#L128-L129
非原生 FP4 KV cache 关闭。KV 和 index cache 均为 FP8(kv_cache_dtype: fp8,index-cache-dtype: fp8)。 recipe agentic.yaml
SSM/KDA/Mamba 状态 cache 低于 FP32 不适用。MiniMax-M3.1-Flash 没有 SSM、KDA 或 Mamba 状态。 模型 config

以上优化均未使用,因此不需要三组对照的资格评测。PR 及其关联 issue 中没有模型训练方的否决。精度检查:GSM8K strict exact match 在 C8 为 0.9666,C16 为 0.9629,C48 为 0.9644,均为 1,319 题。

@seungrokj seungrokj left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Signed: seungrokj

@seungrokj

seungrokj commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

FYI. amd/MiniMax-M3.1-Flash-Quark-MXFP4 is already used in minimaxm31flash-fp4-mi355x-vllm-agentic-dspark:

@SemiAnalysisAI SemiAnalysisAI deleted a comment from github-actions Bot Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Verdict: WARN ⚠️

⚠️ Pareto coverage needs additional review. At least 5 points per affected throughput-versus-E2EL frontier are highly recommended. Below 5, or when coverage cannot be verified, merge only with an explicit, recorded admin bypass for the assessed commit; this advisory comment does not grant or enforce a bypass.

⚠️ Check 14 (Pareto coverage): WARN — curve minimaxm31flash / agentic-coding / MI355X ATOM / fp4 / P90 E2EL has 3/5 measured frontier points (c4 13.42 s 8,233 tok/s/GPU; c8 22.65 s 10,650; c32 22.78 s 47,248); c1, c2, c16 and c48 are dominated. Evidence: results_bmk of run 42457076638 attempt 1 on 83c37679e, image rocm/atom-dev:nightly_202611060646@sha256:857297cc…. No admin bypass is recorded on the PR; an explicit admin exception for 520abf1f and this curve is required before merge.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — seungrokj is a named owner of inferencex-e2e/configs/amd-master.yaml and of srt-slurm-recipes/**/mi355x-*/; perf-changelog.yaml falls under the catch-all.

✅ Check 1 (In-PR sweep + evals): PASS — in-PR commit 83c37679e has executed, successful agentic / (c1-c48) and agentic eval / (c8, c16, c48) jobs in run 42457076638. The later commits are not material: 520abf1f changes only comments and changelog text for this key, and merge 7886b82a brings in only other keys, docs and result staging.

✅ Check 2 (Evals): PASS — GSM8K strict on the PR image: C8 0.9666, C16 0.9629, C48 0.9644 (n=1319). All rows are within 2 sigma of the best sibling and of the vLLM GB300 cohort (run 41552552280, 0.9659). No conditional backend change needs a path proof.

➖ Check 3 (Upstream recipe): N/A — the only affected config is framework: atom and no vLLM/SGLang serving config changes. For information: the linked ROCm/ATOM#2747 is still OPEN, and ATOM main's recipes/MiniMax-M3-Agentic-InferenceX.md has no M3.1-Flash section yet.

✅ Check 4 (Reuse command): PASS — COLLABORATOR zhuyuhua-v posted /use 42457076638 and COLLABORATOR seungrokj posted /reuse-sweep-run 42457076638, each on its own line; both name the Check 1 run.

✅ Check 5 (Checklist template): PASS — every item of the current PR_REVIEW_CHECKLIST.md template is present and checked.

✅ Check 6 (Upstream images / ordering): PASS — (a) N/A for ATOM; (b) vLLM and SGLang entries for minimaxm31flash on cluster:mi355x-amds already exist (minimaxm31flash-fp4-mi355x-vllm-agentic-dspark, -sglang-agentic-dspark); (c) the vLLM/SGLang main-branch rule does not apply, and the image's ATOM e1295074f is an ancestor of ATOM main anyway.

✅ Check 7 (Deprecated models): PASS — minimaxm31flash agentic coding is active in MODELS.md (review date 2026-11-09).

✅ Check 8 (Architecture hacks): PASS — no hf-overrides, online quantization, or indexer sharing. FP8 KV is used, but not to offset another quality trade. minimaxm31flash is in the integer all-reduce baseline matrix (grandfathered), and AITER_QUICK_REDUCE_QUANTIZATION is unset anyway, which AITER b0e8ce03 treats as NONE (disabled).

✅ Check 9 (Chat templates): PASS — AgentX trace replay drives the server through AIPerf with the default chat endpoint and conversational traces.

✅ Check 10 (Engine patches): PASS — no reachable patching: no patch files, setup scripts or source edits in the recipe; the run log shows a stock python3 -m atom.entrypoints.openai_server launch.

✅ Check 11 (Agentic golden AL): PASS — the shared synthetic_acceptance.py injects --spec-decode-acceptance-length 4.04 for every throughput point (seen in the c48 job log). That matches the golden minimax-m3.1-flash__dspark__native__probabilistic-block thinking_on value at K7. Evals keep real acceptance.

➖ Check 12 (Append-only): N/A — the changelog entry has no append-only: true.

✅ Check 13 (Draft precision): PASS — the draft path MiniMaxAI@2f33a8b9/dspark has the same blob and LFS oids as amd/MiniMax-M3.1-Flash-Quark-MXFP4 dspark/. It is stored as mxfp8 [1,32], with fc, Markov and confidence heads left in BF16, and ATOM dspark.py builds the draft's quantization from the draft's own config. There are no draft overrides, SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set, and FP8 draft KV is inherited from the target.

✅ Check 15 (PR description): PASS — the image, points, settings, GSM8K scores and throughput/P90 E2EL table match the pinned recipe and run 42457076638 artifacts.

➖ Check 16 (NVIDIA overview): N/A — the PR affects only MI355X (non-NVIDIA).

➖ Check 17 (Metrics publication): N/A — this is an ATOM single-node agg path with no Dynamo TRT-LLM settings.

✅ Check 18 (Recipe/image consistency): PASS — (a) the key generates 7 points at the pinned SHA; (b) TP4, the concurrency lists and LMCache 4x200 GiB (858.99 GB) all fit the 869 GB DRAM budget; (c) the digest resolves; (d)/(e) N/A; (f) the changelog matches the diff; (g) nothing is removed; (h) new key, no deltas; (i) ATOM has no interval knob and streams each engine step, at most 8 tokens with K7, so 20 or fewer.

Assessed commit: 520abf1f0fe4f275fd2f69e025c625760cc1f30a.

@seungrokj

Copy link
Copy Markdown
Collaborator

@zhuyuhua-v is this perf expected and can be merged as it is ?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure AMD full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants