Repository navigation
Conversation
Signed-off-by: Nils Matteson <nils@inferact.ai>
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
Signed-off-by: Nils Matteson <nils@inferact.ai>
|
|
||
| from infx.bench.env import BenchError | ||
|
|
||
| HARNESS = "agentx-harness==1.0.12" |
There was a problem hiding this comment.
removed the duplicate pin. requirements.txt owns the client version now. reuse still checks that the installed version matches and supports the drain check.
| @@ -0,0 +1,576 @@ | |||
| """Opt-in reuse of one ready vLLM server across the points of a same-launch-config group. | |||
There was a problem hiding this comment.
preferably this would be abstracted to a level such that this same functionality could be added for other engines in the future?
There was a problem hiding this comment.
split the vLLM-specific reset, metrics and router handling into a backend selected by the recipe’s engine type. other engines can add a backend without changing the point loop. only vLLM is enabled for now.
Signed-off-by: Nils Matteson <nils@inferact.ai>
Adds opt-in
AGENTX_SERVER_REUSEfor concurrency points with identical server configs. Each point keeps separate artifacts. The next point starts only after a clean client drain and exit, idle engines with continuous counters, and a successful non-preempting local prefix reset. Existing recipes keep their single-point behavior.requirements.txtselects the client version. Reuse validates both the selected version and the installed wheel against the drain parser's supported log formats. vLLM's reset, metrics and router rules now sit inreuse_vllm.py, behind a backend selected by the bound recipe's engine type. Only direct vLLM and vLLM Router are enabled.A live c8 → c16 check on the earlier integrated version passed on two B300s with DeepSeek-V4.1-Flash through vLLM Router. The follow-up changes client validation and the backend boundary; that GPU run does not qualify the refactor itself. Prefix reset leaves JIT and other process state intact, so benchmark equivalence remains a separate acceptance question. Four existing local-suite failures still reproduce as previously recorded on the base revision.
Validation details
Slurm job 8438 completed in 24m20s on two B300s, before the backend/client-validation refactor. It used vLLM image digest
2b9fc4b18d0aaabe4d5cad020d9270106472ebe65f5d3bc02c50526b8d9fd280, vLLM Router 0.1.14, TP1/DP2 and 60-second profiling windows with--unsafe-override.Both points completed every issued warmup/profiling request. The integrated reset used
reset_running_requests=false&reset_external=false. A separate post-group probe checked cold miss, warm hit, reset and first-post-reset miss on each engine. Client sessions, router and engine exited cleanly. The power sidecar was omitted and recorded in the effective launch identity. This established reset mechanics, not performance or accuracy equivalence.The refactor retains the vLLM request paths and reset semantics and separates them from point sequencing. The installed-client check is new. No new engine or benchmark point is enabled. The existing local-suite failures are three revision-subprocess cases and one changelog-dispatch case, previously reproduced on base
4bc1b3dda4df428d2cee984d8af9c8ddffade3ec.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.No existing recipe opts in and no config keys change, so there is no performance changelog entry or requested sweep.
中文
新增按需启用的
AGENTX_SERVER_REUSE,让服务器配置相同的并发测试点复用同一部署。每个点保留独立产物,只有客户端请求全部结束、进程退出、引擎空闲且计数器连续、非抢占式本地前缀缓存重置成功后,才启动下一点。现有配方仍按单点运行。客户端版本由
requirements.txt选择。复用路径会检查配置版本和实际安装的 wheel 是否属于 drain 日志解析器已验证的版本。vLLM 的 reset、metrics 和 router 规则移入reuse_vllm.py,由绑定配方的 engine 类型选择 backend。目前仍只支持 vLLM 直连和 vLLM Router。较早的集成版本在两张 B300 上完成了 DeepSeek-V4.1-Flash 的 c8 → c16 实机检查,两个引擎的缓存清除和进程清理均正常。这次实机结果早于本次 backend 和客户端校验重构,不代表新版本已通过 GPU 验证,也不证明性能或精度等价。前缀重置不会清除 JIT 等进程状态。四个既有本地测试失败仍存在,已在基础版本上复现。
AI model disclosure
gpt-6.1-sol: review-follow-up integration and client-pin comment compatibility.gpt-6-astra: final source review, regression-test correction and publication follow-up.claude-opus-5-5: implementation, backend/client-validation refactor and focused reviews.claude-fable-5-1: reset/protocol and scope review.inferact-k3: early source mapping; the endpoint's underlying model version was not independently verified.