Repository navigation
fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput points - #4840
Conversation
…hput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.
cquil11
added
the
non-canary-full-sweep-enabled
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Besides the inline finding, I checked two other concerns and ruled them out. The long parametrize line and the trailing blank line in the new test won't fail the CI lint job, because infx/ruff.toml sets extend-exclude = ["tests"] and ignores E501. The direct environment["MODEL_PREFIX"] lookup follows the existing required-key access at lines 201 and 222 of the same module.
Extended reasoning...
I ruled out the lint concern because the ruff config excludes the tests directory. I ruled out the MODEL_PREFIX KeyError concern because the module already reads that key directly on the same build_overrides path.
| if environment["MODEL_PREFIX"].startswith("kimi"): | ||
| for key in KIMI_VLLM_PARSER_ARGS: | ||
| if key in (role.get("args") or {}): | ||
| overrides += ["--unset", f"{prefix}.args.{key}"] |
There was a problem hiding this comment.
🔴 Kimi vLLM synthetic throughput benchmarks now run without their parsers, but no perf-changelog entry is added, so the re-run the PR describes won't be scheduled. CONTRIBUTING.md:30 requires a new entry at the end of inferencex-e2e/perf-changelog.yaml for any change to what a benchmark runs. The diff only touches synthetic_acceptance.py and its test. Fix: append one entry listing every affected Kimi vLLM synthetic-acceptance config key, including kimik3.1-fp4-b300-vllm-agentic-dspark. python -m infx.matrix.changed --base origin/main lists them.
Why this was flagged
synthetic_acceptance.py:304-307 now adds --unset roles.<name>.args.{enable-auto-tool-choice,tool-call-parser,reasoning-parser} for every vLLM worker whose MODEL_PREFIX starts with "kimi" and that receives a synthetic acceptance length. That changes the server command, and the PR's own numbers show it changes ITL and error counts. CONTRIBUTING.md:30 says such a PR MUST append a perf-changelog.yaml entry for the affected config keys, and the changelog diff is what selects the sweep. git diff --stat shows only two changed files and no perf-changelog.yaml. So kimik3.1-fp4-b300-vllm-agentic-dspark (nvidia-master.yaml:5727) is not re-run. Its published ITL and interactivity stay at the 17–26% optimistic values from the parsers-on run until some unrelated changelog entry triggers that key.
Verification: The diff touches only inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py and its test, and perf-changelog.yaml is unchanged. Lines 304-307 add --unset {prefix}.args.{key} for the parsers when MODEL_PREFIX starts with "kimi". infx.matrix.changed does not see script changes like this one, so without an entry no sweep or result collection is triggered.
cquil11
removed
the
non-canary-full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42101946320 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42101946320 |
Under synthetic acceptance, vLLM streams draft tokens that were never verified. The
kimi_k3tool and reasoning parsers stop streaming on some of them, so the client gets either an empty response, which AIPerf reports as an error, or one cut short. AIPerf ends E2E latency and ITL at the last content chunk, so the cut-short responses also under-measure both.For Kimi models, this removes
--enable-auto-tool-choice,--tool-call-parserand--reasoning-parserfrom vLLM throughput servers that use synthetic acceptance. Evals and acceptance gates verify drafts for real and keep the parsers.Validation: #4797 B200 run 42071652809, with the same change, against the parsers-on run 41932289426:
Profiling phase only. With parsers off, c24 and c48 each also had one client connection reset.
On the same 971 c8 requests, completion tokens, server decode time and HTTP stream time are unchanged.