Skip to content

Replace optimization bans with eval qualification / 以评估准入替代优化禁令 - #4705

Open
functionstackx wants to merge 6 commits into
mainfrom
docs/eval-qualified-optimizations
Open

functionstackx wants to merge 6 commits into
mainfrom
docs/eval-qualified-optimizations

Conversation

@functionstackx

@functionstackx functionstackx commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks to feedback from AMD engineer, we have modified this rule for upstream inferenceX repo to allow submitors of the technology to prove that technology is valid and doesn't negatively impact quality on an full set of non trival evaluation instead of outright banning an technique that has been shown to negatively impact quality. This also allows us to set the bar that an submitor will truly believe that an optimization is truly useful for production and truly believe that it does not drop quality by requiring the submitor to manually run non trival evaluations. We suspect based on this new guideline, we will be getting a lot less attempted submissions with eyebrow raising optimizations...

Replace the integer all-reduce ban and model-grandfathering exception with reproducible accuracy qualification for reduced-precision all-reduce/reduce-scatter, online linear-layer quantization, shared-expert quantization, reduced-precision Engram weights, non-native cross-indexer sharing, non-native FP4 KV cache, and reductions in SSM/KDA/Mamba state-cache dtype below FP32.

  • Kimi requires the entire KVV evaluation suite. Other models require all of Terminal-Bench 4.0, HLE, and MMMU Pro Vision, or a substitute set agreed upon with core maintainers.
  • The model trainer (for example, DeepSeek or Moonshot) may veto any optimization for its own model if it considers the optimization unsuitable for production use because it reduces evaluation accuracy, even when the required evals pass. Record the attributable objection, rationale, and scope in the PR; only the trainer can withdraw or narrow it. This does not require affirmative trainer approval for every submission.
  • State-cache precision reductions require FP32 SSM/KDA/Mamba state caches in both baseline arms and the proposed lower-precision dtype in the optimized arm, verified separately from weight and KV-cache precision.
  • Engram weight precision reductions, including online or offline quantization, require unreduced Engram weights in both baseline arms and the proposed reduction only in the optimized arm. Verify Engram weight dtype and conversion separately from other model weights.
  • Require raw logs and complete manual reproduction scripts for three matched evaluation arms: original full-precision weights without reduced-precision communications or the tested optimization; the quantized checkpoint without the optimization; and the same quantized checkpoint with it. Report checkpoint, optimization, and total accuracy effects separately.
  • Update MODELS.md, CONTRIBUTING.md, the review checklist, and verifier Checks 2/8 together. Qualified non-native cross-indexer sharing is an explicit exception to the general architecture-FLOP restriction. Enabling image defaults receive the same scrutiny as explicit recipe flags.

This applies to submissions for existing and new models, including image updates, without invalidating historical results. The separate draft-model precision and engine-patch restrictions remain unchanged. No new universal numeric accuracy tolerance is invented.

This changes documentation and the advisory sign-off verifier prompt only. It does not change benchmark recipes, implement the evaluation suites, or add a mechanical merge gate. Related open policy proposal #4510 contains blanket prohibitions that would need reconciliation with this policy.

AI model disclosure
  • Model/version: GPT-6 Astra.
  • Role: Repository inspection, policy drafting, consistency review, and local validation. No delegated agents.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other: advisory sign-off review policy

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries.
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label. Not applicable: no benchmark changes or changelog entry.
  • Before merging via reuse, an authorized maintainer has commented /use <run_id> on this PR after a final green full sweep with evals. Not applicable: this policy-only PR has no sweep artifacts to reuse.
中文

用可复现的准确率评估要求替换整数 all-reduce 禁令及旧模型豁免,覆盖低精度 all-reduce/reduce-scatter、线性层在线量化、共享专家量化、Engram 权重精度降低、非原生跨 indexer 共享、非原生 FP4 KV cache,以及将 SSM/KDA/Mamba 状态缓存 dtype 降至 FP32 以下。

  • Kimi 必须通过完整 KVV 评估套件。其他模型必须通过 Terminal-Bench 4.0、HLE 和 MMMU Pro Vision 全部评估,或经核心维护者同意的替代评估集合。
  • 模型训练方(例如 DeepSeek 或 Moonshot)若认为某项优化会降低评估准确率、不适合生产使用,可否决该优化在其模型上的提交,即使规定的评估已经通过。PR 必须记录可归属到训练方的异议、理由及适用范围;只有训练方可以撤回或缩小否决范围。这不要求每次提交都取得训练方的明确批准。
  • 测试状态缓存精度降低时,两组基线必须使用 FP32 SSM/KDA/Mamba 状态缓存,优化组使用拟提交的较低精度 dtype;状态缓存精度须与权重及 KV-cache 精度分别核验。
  • 测试 Engram 权重精度降低(包括在线或离线量化)时,两组基线须保留未降低精度的 Engram 权重,仅在优化组应用拟提交的精度降低。Engram 权重 dtype 及转换须与其他模型权重分别核验。
  • 必须提供原始日志和完整的手动复现脚本,覆盖三组条件一致的评估:原始全精度权重,不使用低精度通信或待测优化;量化权重检查点,不启用待测优化;同一量化检查点,启用待测优化。分别报告检查点量化、待测优化及两者叠加的准确率影响。
  • 同步更新 MODELS.md、CONTRIBUTING.md、审查清单和验证器 Check 2/8。通过上述评估的非原生跨 indexer 共享是架构 FLOP 限制的明确例外。镜像默认启用的优化与显式配置接受相同审查。

要求适用于新旧模型的提交,包括镜像更新,不追溯撤销历史结果。保留独立的 draft-model 精度和引擎补丁限制,不新增统一的准确率数值容差。

本 PR 只修改文档及建议性签核验证器提示词,不修改基准测试配置、不实现评估套件,也不新增强制合并门禁。仍在审查中的 #4510 包含一律禁止相关优化的条款,需要与本政策协调。

AI 模型:GPT-6 Astra,负责仓库检查、政策起草、一致性审查和本地验证;未委派其他智能体。

Created with Perplexity

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment on lines +47 to +49
1. Full-precision weights, without reduced-precision communications and with the optimization under test disabled. Use the original full-precision checkpoint, not a quantized checkpoint cast back to a wider dtype.
2. The quantized-weight checkpoint, with the optimization under test disabled.
3. The same quantized-weight checkpoint as arm 2, with the optimization under test enabled.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Kimi-K3 submitters cannot qualify their INT4 quick-reduce recipes, because the required arm 1 checkpoint does not exist. The original moonshotai/Kimi-K3 release is already FP4. Arm 1 needs an "original full-precision checkpoint" and rules out casting a quantized one. Arm 2 assumes a separate quantized checkpoint, but for natively low-precision releases it is the same checkpoint. Fix: define arm 1 as the vendor's original release checkpoint at its shipped precision, and allow arms 1 and 2 to be the same checkpoint when nothing else ships. Apply the same change to the verifier prompt at line 475 and to the checklist.

Why this was flagged

The moonshotai/Kimi-K3 checkpoint used in inferencex-e2e/configs/nvidia-master.yaml:1288 has precision: fp4. Several Kimi-K3 recipes already set INT4 quick reduce, for example kimik3/vllm/mi355x-fp4-mtp/agentic.yaml:57 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: INT4. MODELS.md:47 makes arm 1 the "original full-precision checkpoint, not a quantized checkpoint cast back to a wider dtype", and codeowner-signoff-verify-prompt.md:475 repeats that rule. Moonshot has published no wider-precision Kimi-K3 checkpoint, so arm 1 cannot be produced. Under the new Check 8(c), missing evidence means FAIL. As a result, every recipe or image update to these configs fails sign-off, and the new grandfathering removal means no exemption applies. That turns the PR's "qualification, not automatic rejection" into a de facto ban for natively quantized models.

Verification: The new rule at inferencex-e2e/docs/MODELS.md:47 defines arm 1 as: "Full-precision weights ... Use the original full-precision checkpoint, not a quantized checkpoint cast back to a wider dtype." For Kimi-K3, every config in the repo uses the vendor's own repo moonshotai/Kimi-K3 with precision: fp4. No other Kimi-K3 weight checkpoint appears anywhere in the repo.


### Required evaluations

- Kimi models must pass the entire KVV evaluation suite.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Kimi submitters and the sign-off verifier have no definition of "the entire KVV evaluation suite", and a vacuous existing CI job can pass for it. The acronym appears nowhere else, has no link and no pinned version. The repo's existing kimi-vendor job kimi_tool_call_schema_full is called "full" but only checks tool-call schemas, with a 0.0 threshold. Fix: in MODELS.md and Check 8(c), link and pin KVV (upstream commit), list the benchmarks it must include, and state its pass criteria. Also state whether kimi_tool_call_schema_full alone counts.

Why this was flagged

A Kimi recipe with INT4 quick-reduce (kimik3/vllm/mi355x-fp4-mtp/agentic.yaml:57) now needs "the entire KVV evaluation suite" (MODELS.md:38, codeowner-signoff-verify-prompt.md:467, CONTRIBUTING.md:42). None of these defines KVV, links it or pins a version, even though non-Kimi suites are pinned by version ("Terminal-Bench 4.0"). The repo already runs MoonshotAI/Kimi-Vendor-Verifier at commit 3dad65a760a8867cda72f6dd8848d876a4e851b4. It runs as the automatic kimi_tool_call_schema_full task on every Kimi K3 AgentX point (infx/evals/EVALS.md:162-191, 257-280). That task only covers the 204 tool-call schema cases. Its threshold is 0.0, so any completed score passes. A submitter can therefore offer three green runs of that job as the "entire KVV suite", and the verifier has nothing to reject it against. Another verifier may instead demand the whole upstream verifier. The verdict depends on how each reader interprets the term. The base branch had no KVV requirement. The prompt's "do not accept a green job alone" line does not define what the suite contains.

Verification: Trigger: a Kimi submission using a covered optimization (for example INT4 quick-reduce) offers the repo's existing Kimi "full suite" as its KVV evidence. MODELS.md:38 does not expand the acronym, link it, pin a version or list its benchmarks. EVALS.md:162 calls kimi_tool_call_schema_full "The Kimi full suite". thresholds.yaml:5 sets it to 0.0.

Comment on lines 425 to 433
### (a) No architecture FLOP reduction
Scan for architecture-override knobs: `--hf-overrides`, `hf_overrides`,
`--json-model-override-args`, config-editing `sed`/`jq` on the model files, etc. If
present, determine whether the override changes computed FLOPs vs the model's native
config and whether the model natively supports that mode. FAIL with the exact
flag/value if architecture FLOPs are reduced without native model support.
config and whether the model natively supports that mode. Assess non-native
cross-indexer sharing under (c); do not reject it solely for reducing indexer FLOPs
if it qualifies there. FAIL with the exact flag/value for other architecture-FLOP
reductions without native model support.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Indexer overrides now get unpredictable sign-off verdicts. The docs allow qualified "cross-indexer sharing" but still ban "arbitrary indexer skipping". Neither term is defined. The repo's own banned example, use_index_cache/index_topk_freq, reuses another layer's top-k, so it fits both labels. Check 8(b) puts these knobs and skip_index under one heading, "indexer skipping or sharing". Fix: define sharing operationally, for example reusing an earlier layer's indexer top-k. Say which knobs and override values fall in each bucket. Apply the definition the same way in MODELS.md, Check 8 and the checklist.

Why this was flagged

A PR adds --hf-overrides setting use_index_cache/index_topk_freq on a model with no native index sharing. Earlier PRs removed this exact override, and the changelog calls it an "indexer-skipping override" and a "cross-layer indexer-skip" (perf-changelog.yaml:4486, 4760). Check 8(a) at codeowner-signoff-verify-prompt.md:429-433 sends "non-native cross-indexer sharing" to (c), where it can PASS with qualification. Any "other architecture-FLOP reductions" FAIL outright, and lines 417-419 forbid "arbitrary indexer skipping". No line says which bucket this override belongs to. The same split appears in MODELS.md:31,57 and PR_REVIEW_CHECKLIST.md:63. One verifier run can FAIL a qualified submission while another PASSes an indexer skip by calling it sharing. On the base branch the override always FAILed Check 8(a), so the verdict was predictable.

Verification: Trigger: a PR adds --hf-overrides with use_index_cache/index_topk_freq on a model that has no native index sharing. 8(b) (codeowner-signoff-verify-prompt.md:435-436) still lists index_topk_freq, use_index_cache and skip_index under one heading, "indexer skipping or sharing". None of the changed files defines "sharing" or "skipping", or maps the knobs to either term.

@zhuyuhua-v

Copy link
Copy Markdown
Collaborator

Hi @functionstackx I was wondering if there are plans to provide explicit test scripts for the evaluation methods mentioned here (such as HLE and Terminal-Bench 4.0) in the future?
Currently, I couldn't quite find clear instructions on how to test the model following the recommended approach and requirements outlined in the PR...

@functionstackx

Copy link
Copy Markdown
Collaborator Author

@zhuyuhua-v how does the AMD HLE and TB 4.0 look like? What configs do y'all use?

@zhuyuhua-v

Copy link
Copy Markdown
Collaborator

Hi @functionstackx, here's what we've run HLE so far on MiniMax-M3.1-Flash (MI355X, TP4). .

HLE setup

  • Dataset: cais/hle @ 5a81a4c, text-only subset (image == ""). That's 2,158 questions, the same set as the "HLE (without tools)" row on the model card (43.4). https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash
  • Prompts: the official system prompts from centerforaisafety/hle (exact-answer / multiple-choice formats), with the question as the user turn.
  • Sampling: the card's recommended temperature=1.0, top_p=0.95. Thinking enabled, reasoning_effort=max (the card doesn't say which effort its HLE number used). max_tokens=262144, one sample per question. Speculative decoding uses real acceptance, not the synthetic golden AL.
  • Judge: the official HLE judge prompt with structured output, but judged by Claude Opus 5.5 instead of o3-mini (we don't have o3-mini access). Only the final content is judged; empty or truncated answers count as incorrect. About 1% of items were declined by the judge and re-judged with Claude Opus 4.8 / Sonnet 5.5. One item per arm stayed unjudged and is counted incorrect.
  • Serving: ROCm/ATOM f328ca296 , aiter with ROCm/aiter #7056/#7057/#7060/#7061, image rocm/atom-dev:nightly_202610261558. Checkpoint amd/MiniMax-M3.1-Flash-Quark-MXFP4, TP4, DSpark K=7, MXFP4 sparse KV + MXFP4 index cache, max-model-len 1M, 128 concurrent requests.

Results

correct score 95% CI
927 / 2158 42.96% [40.9, 45.1]

A few questions on the policy

  1. Is a Claude judge acceptable, or do you want o3-mini specifically? Do you have a preferred reasoning_effort and max_tokens for HLE?
  2. Do you plan to publish a reference harness with pinned versions for HLE, TB 4.0 and MMMU Pro Vision, so every submitter runs the same thing?

@functionstackx

Copy link
Copy Markdown
Collaborator Author

@zhuyuhua-v wuay about for the 4.0, do u have results for that?

My thinking is that the submitors will bring their own eval harness and manual reprod script initially and then eventually one of them will have an high quality reprod script that we can adopt for everyone.

Would u like to help make an high quality reprod script ?

…ghts

Co-authored-by: Perplexity <noreply@perplexity.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants