Repository navigation
Replace optimization bans with eval qualification / 以评估准入替代优化禁令 - #4705
functionstackx wants to merge 6 commits into
Conversation
| 1. Full-precision weights, without reduced-precision communications and with the optimization under test disabled. Use the original full-precision checkpoint, not a quantized checkpoint cast back to a wider dtype. | ||
| 2. The quantized-weight checkpoint, with the optimization under test disabled. | ||
| 3. The same quantized-weight checkpoint as arm 2, with the optimization under test enabled. |
There was a problem hiding this comment.
🔴 Kimi-K3 submitters cannot qualify their INT4 quick-reduce recipes, because the required arm 1 checkpoint does not exist. The original moonshotai/Kimi-K3 release is already FP4. Arm 1 needs an "original full-precision checkpoint" and rules out casting a quantized one. Arm 2 assumes a separate quantized checkpoint, but for natively low-precision releases it is the same checkpoint. Fix: define arm 1 as the vendor's original release checkpoint at its shipped precision, and allow arms 1 and 2 to be the same checkpoint when nothing else ships. Apply the same change to the verifier prompt at line 475 and to the checklist.
Why this was flagged
The moonshotai/Kimi-K3 checkpoint used in inferencex-e2e/configs/nvidia-master.yaml:1288 has precision: fp4. Several Kimi-K3 recipes already set INT4 quick reduce, for example kimik3/vllm/mi355x-fp4-mtp/agentic.yaml:57 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: INT4. MODELS.md:47 makes arm 1 the "original full-precision checkpoint, not a quantized checkpoint cast back to a wider dtype", and codeowner-signoff-verify-prompt.md:475 repeats that rule. Moonshot has published no wider-precision Kimi-K3 checkpoint, so arm 1 cannot be produced. Under the new Check 8(c), missing evidence means FAIL. As a result, every recipe or image update to these configs fails sign-off, and the new grandfathering removal means no exemption applies. That turns the PR's "qualification, not automatic rejection" into a de facto ban for natively quantized models.
Verification: The new rule at inferencex-e2e/docs/MODELS.md:47 defines arm 1 as: "Full-precision weights ... Use the original full-precision checkpoint, not a quantized checkpoint cast back to a wider dtype." For Kimi-K3, every config in the repo uses the vendor's own repo moonshotai/Kimi-K3 with precision: fp4. No other Kimi-K3 weight checkpoint appears anywhere in the repo.
|
|
||
| ### Required evaluations | ||
|
|
||
| - Kimi models must pass the entire KVV evaluation suite. |
There was a problem hiding this comment.
🟡 (optional) Kimi submitters and the sign-off verifier have no definition of "the entire KVV evaluation suite", and a vacuous existing CI job can pass for it. The acronym appears nowhere else, has no link and no pinned version. The repo's existing kimi-vendor job kimi_tool_call_schema_full is called "full" but only checks tool-call schemas, with a 0.0 threshold. Fix: in MODELS.md and Check 8(c), link and pin KVV (upstream commit), list the benchmarks it must include, and state its pass criteria. Also state whether kimi_tool_call_schema_full alone counts.
Why this was flagged
A Kimi recipe with INT4 quick-reduce (kimik3/vllm/mi355x-fp4-mtp/agentic.yaml:57) now needs "the entire KVV evaluation suite" (MODELS.md:38, codeowner-signoff-verify-prompt.md:467, CONTRIBUTING.md:42). None of these defines KVV, links it or pins a version, even though non-Kimi suites are pinned by version ("Terminal-Bench 4.0"). The repo already runs MoonshotAI/Kimi-Vendor-Verifier at commit 3dad65a760a8867cda72f6dd8848d876a4e851b4. It runs as the automatic kimi_tool_call_schema_full task on every Kimi K3 AgentX point (infx/evals/EVALS.md:162-191, 257-280). That task only covers the 204 tool-call schema cases. Its threshold is 0.0, so any completed score passes. A submitter can therefore offer three green runs of that job as the "entire KVV suite", and the verifier has nothing to reject it against. Another verifier may instead demand the whole upstream verifier. The verdict depends on how each reader interprets the term. The base branch had no KVV requirement. The prompt's "do not accept a green job alone" line does not define what the suite contains.
Verification: Trigger: a Kimi submission using a covered optimization (for example INT4 quick-reduce) offers the repo's existing Kimi "full suite" as its KVV evidence. MODELS.md:38 does not expand the acronym, link it, pin a version or list its benchmarks. EVALS.md:162 calls kimi_tool_call_schema_full "The Kimi full suite". thresholds.yaml:5 sets it to 0.0.
| ### (a) No architecture FLOP reduction | ||
| Scan for architecture-override knobs: `--hf-overrides`, `hf_overrides`, | ||
| `--json-model-override-args`, config-editing `sed`/`jq` on the model files, etc. If | ||
| present, determine whether the override changes computed FLOPs vs the model's native | ||
| config and whether the model natively supports that mode. FAIL with the exact | ||
| flag/value if architecture FLOPs are reduced without native model support. | ||
| config and whether the model natively supports that mode. Assess non-native | ||
| cross-indexer sharing under (c); do not reject it solely for reducing indexer FLOPs | ||
| if it qualifies there. FAIL with the exact flag/value for other architecture-FLOP | ||
| reductions without native model support. | ||
|
|
There was a problem hiding this comment.
🟡 (optional) Indexer overrides now get unpredictable sign-off verdicts. The docs allow qualified "cross-indexer sharing" but still ban "arbitrary indexer skipping". Neither term is defined. The repo's own banned example, use_index_cache/index_topk_freq, reuses another layer's top-k, so it fits both labels. Check 8(b) puts these knobs and skip_index under one heading, "indexer skipping or sharing". Fix: define sharing operationally, for example reusing an earlier layer's indexer top-k. Say which knobs and override values fall in each bucket. Apply the definition the same way in MODELS.md, Check 8 and the checklist.
Why this was flagged
A PR adds --hf-overrides setting use_index_cache/index_topk_freq on a model with no native index sharing. Earlier PRs removed this exact override, and the changelog calls it an "indexer-skipping override" and a "cross-layer indexer-skip" (perf-changelog.yaml:4486, 4760). Check 8(a) at codeowner-signoff-verify-prompt.md:429-433 sends "non-native cross-indexer sharing" to (c), where it can PASS with qualification. Any "other architecture-FLOP reductions" FAIL outright, and lines 417-419 forbid "arbitrary indexer skipping". No line says which bucket this override belongs to. The same split appears in MODELS.md:31,57 and PR_REVIEW_CHECKLIST.md:63. One verifier run can FAIL a qualified submission while another PASSes an indexer skip by calling it sharing. On the base branch the override always FAILed Check 8(a), so the verdict was predictable.
Verification: Trigger: a PR adds --hf-overrides with use_index_cache/index_topk_freq on a model that has no native index sharing. 8(b) (codeowner-signoff-verify-prompt.md:435-436) still lists index_topk_freq, use_index_cache and skip_index under one heading, "indexer skipping or sharing". None of the changed files defines "sharing" or "skipping", or maps the knobs to either term.
|
Hi @functionstackx I was wondering if there are plans to provide explicit test scripts for the evaluation methods mentioned here (such as HLE and Terminal-Bench 4.0) in the future? |
|
@zhuyuhua-v how does the AMD HLE and TB 4.0 look like? What configs do y'all use? |
|
Hi @functionstackx, here's what we've run HLE so far on MiniMax-M3.1-Flash (MI355X, TP4). . HLE setup
Results
A few questions on the policy
|
|
@zhuyuhua-v wuay about for the 4.0, do u have results for that? My thinking is that the submitors will bring their own eval harness and manual reprod script initially and then eventually one of them will have an high quality reprod script that we can adopt for everyone. Would u like to help make an high quality reprod script ? |
…ghts Co-authored-by: Perplexity <noreply@perplexity.ai>
Thanks to feedback from AMD engineer, we have modified this rule for upstream inferenceX repo to allow submitors of the technology to prove that technology is valid and doesn't negatively impact quality on an full set of non trival evaluation instead of outright banning an technique that has been shown to negatively impact quality. This also allows us to set the bar that an submitor will truly believe that an optimization is truly useful for production and truly believe that it does not drop quality by requiring the submitor to manually run non trival evaluations. We suspect based on this new guideline, we will be getting a lot less attempted submissions with eyebrow raising optimizations...
Replace the integer all-reduce ban and model-grandfathering exception with reproducible accuracy qualification for reduced-precision all-reduce/reduce-scatter, online linear-layer quantization, shared-expert quantization, reduced-precision Engram weights, non-native cross-indexer sharing, non-native FP4 KV cache, and reductions in SSM/KDA/Mamba state-cache dtype below FP32.
MODELS.md,CONTRIBUTING.md, the review checklist, and verifier Checks 2/8 together. Qualified non-native cross-indexer sharing is an explicit exception to the general architecture-FLOP restriction. Enabling image defaults receive the same scrutiny as explicit recipe flags.This applies to submissions for existing and new models, including image updates, without invalidating historical results. The separate draft-model precision and engine-patch restrictions remain unchanged. No new universal numeric accuracy tolerance is invented.
This changes documentation and the advisory sign-off verifier prompt only. It does not change benchmark recipes, implement the evaluation suites, or add a mechanical merge gate. Related open policy proposal #4510 contains blanket prohibitions that would need reconciliation with this policy.
AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries.inferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label. Not applicable: no benchmark changes or changelog entry./use <run_id>on this PR after a final green full sweep with evals. Not applicable: this policy-only PR has no sweep artifacts to reuse.中文
用可复现的准确率评估要求替换整数 all-reduce 禁令及旧模型豁免,覆盖低精度 all-reduce/reduce-scatter、线性层在线量化、共享专家量化、Engram 权重精度降低、非原生跨 indexer 共享、非原生 FP4 KV cache,以及将 SSM/KDA/Mamba 状态缓存 dtype 降至 FP32 以下。
MODELS.md、CONTRIBUTING.md、审查清单和验证器 Check 2/8。通过上述评估的非原生跨 indexer 共享是架构 FLOP 限制的明确例外。镜像默认启用的优化与显式配置接受相同审查。要求适用于新旧模型的提交,包括镜像更新,不追溯撤销历史结果。保留独立的 draft-model 精度和引擎补丁限制,不新增统一的准确率数值容差。
本 PR 只修改文档及建议性签核验证器提示词,不修改基准测试配置、不实现评估套件,也不新增强制合并门禁。仍在审查中的 #4510 包含一律禁止相关优化的条款,需要与本政策协调。
AI 模型:GPT-6 Astra,负责仓库检查、政策起草、一致性审查和本地验证;未委派其他智能体。
Created with Perplexity