Skip to content

feat(agentx): reuse servers across compatible points / feat(agentx): 在兼容测试点间复用服务器 - #4743

Open
matteso1 wants to merge 3 commits into
SemiAnalysisAI:mainfrom
matteso1:matteso1/agentx-server-reuse
Open

matteso1 wants to merge 3 commits into
SemiAnalysisAI:mainfrom
matteso1:matteso1/agentx-server-reuse

Conversation

@matteso1

@matteso1 matteso1 commented Nov 2, 2026 •

Copy link
Copy Markdown

Adds opt-in AGENTX_SERVER_REUSE for concurrency points with identical server configs. Each point keeps separate artifacts. The next point starts only after a clean client drain and exit, idle engines with continuous counters, and a successful non-preempting local prefix reset. Existing recipes keep their single-point behavior.

requirements.txt selects the client version. Reuse validates both the selected version and the installed wheel against the drain parser's supported log formats. vLLM's reset, metrics and router rules now sit in reuse_vllm.py, behind a backend selected by the bound recipe's engine type. Only direct vLLM and vLLM Router are enabled.

A live c8 → c16 check on the earlier integrated version passed on two B300s with DeepSeek-V4.1-Flash through vLLM Router. The follow-up changes client validation and the backend boundary; that GPU run does not qualify the refactor itself. Prefix reset leaves JIT and other process state intact, so benchmark equivalence remains a separate acceptance question. Four existing local-suite failures still reproduce as previously recorded on the base revision.

Validation details

Slurm job 8438 completed in 24m20s on two B300s, before the backend/client-validation refactor. It used vLLM image digest 2b9fc4b18d0aaabe4d5cad020d9270106472ebe65f5d3bc02c50526b8d9fd280, vLLM Router 0.1.14, TP1/DP2 and 60-second profiling windows with --unsafe-override.

Both points completed every issued warmup/profiling request. The integrated reset used reset_running_requests=false&reset_external=false. A separate post-group probe checked cold miss, warm hit, reset and first-post-reset miss on each engine. Client sessions, router and engine exited cleanly. The power sidecar was omitted and recorded in the effective launch identity. This established reset mechanics, not performance or accuracy equivalence.

The refactor retains the vLLM request paths and reset semantics and separates them from point sequencing. The installed-client check is new. No new engine or benchmark point is enabled. The existing local-suite failures are three revision-subprocess cases and one changelog-dispatch case, previously reproduced on base 4bc1b3dda4df428d2cee984d8af9c8ddffade3ec.

PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

No existing recipe opts in and no config keys change, so there is no performance changelog entry or requested sweep.

中文

新增按需启用的 AGENTX_SERVER_REUSE,让服务器配置相同的并发测试点复用同一部署。每个点保留独立产物,只有客户端请求全部结束、进程退出、引擎空闲且计数器连续、非抢占式本地前缀缓存重置成功后,才启动下一点。现有配方仍按单点运行。

客户端版本由 requirements.txt 选择。复用路径会检查配置版本和实际安装的 wheel 是否属于 drain 日志解析器已验证的版本。vLLM 的 reset、metrics 和 router 规则移入 reuse_vllm.py,由绑定配方的 engine 类型选择 backend。目前仍只支持 vLLM 直连和 vLLM Router。

较早的集成版本在两张 B300 上完成了 DeepSeek-V4.1-Flash 的 c8 → c16 实机检查,两个引擎的缓存清除和进程清理均正常。这次实机结果早于本次 backend 和客户端校验重构,不代表新版本已通过 GPU 验证,也不证明性能或精度等价。前缀重置不会清除 JIT 等进程状态。四个既有本地测试失败仍存在,已在基础版本上复现。

AI model disclosure
  • Earlier Codex sessions: implementation, integration and validation; their exact model identifiers were not exposed.
  • gpt-6.1-sol: review-follow-up integration and client-pin comment compatibility.
  • gpt-6-astra: final source review, regression-test correction and publication follow-up.
  • claude-opus-5-5: implementation, backend/client-validation refactor and focused reviews.
  • claude-fable-5-1: reset/protocol and scope review.
  • inferact-k3: early source mapping; the endpoint's underlying model version was not independently verified.

Signed-off-by: Nils Matteson <nils@inferact.ai>
@matteso1
matteso1 requested a review from a team November 2, 2026 04:08

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +1871 -17

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

Signed-off-by: Nils Matteson <nils@inferact.ai>

from infx.bench.env import BenchError

HARNESS = "agentx-harness==1.0.12"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try not to hardcode

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed the duplicate pin. requirements.txt owns the client version now. reuse still checks that the installed version matches and supports the drain check.

@@ -0,0 +1,576 @@
"""Opt-in reuse of one ready vLLM server across the points of a same-launch-config group.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

preferably this would be abstracted to a level such that this same functionality could be added for other engines in the future?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

split the vLLM-specific reset, metrics and router handling into a backend selected by the recipe’s engine type. other engines can add a backend without changing the point loop. only vLLM is enabled for now.

Signed-off-by: Nils Matteson <nils@inferact.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants