Repository navigation
Run the AL gate on the agentic eval's own server / 在 AgentX 评测自身服务上运行 AL 门禁 - #4810
Conversation
… AL 门禁 The report-only acceptance-length gate ran in separate jobs, each with its own server launch. It now runs inside the agentic eval job: srt-slurm runs benchmarks/srt_al_gate.sh as the benchmark stage, the only stage that receives the workers' metrics endpoints, then the eval as its post-benchmark eval on the same server. The job is submitted with EVAL_ONLY=false RUN_EVAL=true and telemetry off; srt_eval.sh restores EVAL_ONLY=true for the eval. The shim always exits 0. The planner picks each resolved variant's curve with the run's verifier, as synthetic acceptance does. A gate that cannot run is recorded as skipped with its reason. Dynamo TRT-LLM and llm-d evals are gated. al-gate-prompts.yml becomes callable so run-sweep and e2e-tests hand cached prompts to the agentic eval jobs; a prompts failure leaves the evals ungated and the sweep green. collect-evals summarizes the gate results. DeepSeek-V4.1-Flash and Muse Spark curves gain collection blocks; encoding_dsv41.py copies vLLM's V4.1 encoder from the week of that collection. The gate tests drive a real Runtime and its bootstrap. Removed: the al-gate-select, al-gate, al-gate-multinode and al-gate-collect jobs, the templates' gate-only inputs and steps, the launcher's AL_GATE request fields and validator, al_gate_reuse.py and the gate-job filter in calc_success_rate.py. 仅作报告的接受长度门禁此前在单独作业中运行，每个作业各自启动服务。现在门禁在 AgentX 评测作业内运行：srt-slurm 先以基准阶段运行 benchmarks/srt_al_gate.sh（只有该阶段能拿到 各 worker 的指标端点），再在同一服务上运行基准后评测。作业以 EVAL_ONLY=false RUN_EVAL=true 提交并关闭遥测；srt_eval.sh 为评测恢复 EVAL_ONLY=true。脚本总是以 0 退出。 规划器按运行的验证器为每个解析后的变体选择曲线，与合成接受一致。无法运行的门禁以 skipped 记录原因。Dynamo TRT-LLM 与 llm-d 评测会执行门禁。al-gate-prompts.yml 可被调用， run-sweep 与 e2e-tests 借此把缓存的提示词交给评测作业；提示词准备失败时评测不做门禁， 扫描保持成功。collect-evals 汇总门禁结果。 DeepSeek-V4.1-Flash 与 Muse Spark 曲线新增 collection 块；encoding_dsv41.py 复制了采集 当周 vLLM 的 V4.1 编码器。门禁测试改用真实 Runtime 及其 bootstrap。 移除：al-gate-select、al-gate、al-gate-multinode、al-gate-collect 作业，模板的门禁专用输入 与步骤，启动器的 AL_GATE 请求字段与校验，al_gate_reuse.py，以及 calc_success_rate.py 中的 门禁作业过滤。 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
whats the TLDR of this? that it runs AL gate evals on the gsm8k evals server
so that it doesnt need to waste gpu time restarting the server?
if so, this is W
@functionstackx yes, plus it address some other unresolved issued from #4654. For example:
|
…nto al-gate-on-eval-server # Conflicts: # inferencex-e2e/infx/launch/drivers/srt/__init__.py # inferencex-e2e/infx/launch/drivers/srt/submit.py # inferencex-e2e/infx/tests/launch/test_srt_driver.py
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
inferencex-e2e/benchmarks/srt_al_gate.sh— If the gate replay stalls or runs slowly, the agentic eval never starts. Their shared Slurm job can hit its time limit, so the eval fails. The gate now runs as the eval job's benchmark stage with no time bound, and srt-slurm starts the eval only after that stage exits. Fix: put a wall-clock cap on the gate, e.g.timeoutin the shim or a request timeout plus an overall deadline for the AIPerf replay. When the cap is hit, write theerrorresult and exit 0 so the eval still runs. [also at: inferencex-e2e/infx/bench/al_gate.py:138 - A slow or stalled gate replay now uses up the agentic eval job's own Slurm time, so a report-only gate can make the eval time out.]Why this was flagged
Trigger: an agentic speculative eval whose gate was planned (synthetic_acceptance.py:404-408). srt-slurm runs
bash /infmax-workspace/benchmarks/srt_al_gate.shas the benchmark stage, then the post-benchmark eval only after it exits (srt_al_gate.sh:2-3). The shim callspython3 -m infx.bench al-gatewith notimeout._measure(infx/bench/al_gate.py:87-95) can first bootstrap the AIPerf runtime. It then replays 80 prompts at up to 4096 output tokens throughproc.tee(argv, ...)(al_gate.py:138). The aiperf argv sets no request timeout and nothing bounds the stage. A stuck request, a slow drafter or a slow bootstrap eats the eval job's Slurm time, which was sized for the eval alone. If the job is killed, there is no eval result and the eval job fails. The always-exit-0 guard does not help, because it only covers a gate that exits. On the base branch the gate ran in its own job, so a hang there only cost the gate (the job step usedcontinue-on-error).Verification: srt_al_gate.sh:2-3 says "srt-slurm runs the post-benchmark eval only after a zero exit".
proc.tee(infx/bench/proc.py:68-85) reads the child's output until EOF and has no timeout. Before this change the gate ran in its own separate jobs, so it could not consume an eval's time. Theexit 0handling covers failures but not hangs.
The gate runs as the eval job's benchmark stage, and srt-slurm starts the eval only after it exits, so a stalled replay could use up the job's time limit. The shim now runs the gate under timeout (AL_GATE_TIMEOUT_S, default 1800 s), which signals the whole process group, AIPerf included. The gate turns SIGTERM into an error result and the shim still exits 0. A sweep's call to al-gate-prompts.yml is capped at 30 minutes and continues on error, so a stalled prompts job releases the agentic evals, which then run without the gate. 门禁作为评测作业的基准阶段运行，srt-slurm 在其退出后才开始评测，因此卡住的重放可能耗尽 作业时限。脚本现在用 timeout 运行门禁（AL_GATE_TIMEOUT_S，默认 1800 秒），超时信号会发给 整个进程组（包括 AIPerf）。门禁把 SIGTERM 记录为 error 结果，脚本仍以 0 退出。扫描调用 al-gate-prompts.yml 时限 30 分钟且出错继续，提示词作业卡住时 AgentX 评测会在无门禁的情况下 继续运行。 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
[by Claude Code]
Summary
AgentX throughput forces each speculative deployment's acceptance to its golden AL, and evals cannot see draft quality (speculative decoding is lossless), so a weak drafter goes unnoticed. #4654 measures real AL in separate gate jobs, each with its own server launch. This PR moves that gate into the agentic eval jobs we already run: one server launch serves both the gate and the eval, and the separate gate jobs are removed. The gate stays report-only, so it never fails an eval or blocks a submission.
benchmarks/srt_al_gate.shas srt-slurm's benchmark stage, then the eval as its post-benchmark eval, in the same Slurm job. The benchmark stage is the only one that receives the workers' metrics endpoints. The job is submitted withEVAL_ONLY=false RUN_EVAL=trueand telemetry off;srt_eval.shrestoresEVAL_ONLY=truefor the eval. The server keeps real verification, as eval-only servers already do.infx.bench al-gatereplays the curve's recorded SPEED-Bench collection at concurrency 64, readsAL = 1 + accepted / verificationsfrom the engine's own Prometheus counters, and writesal_gate_<RESULT_FILENAME>.jsonwithpass(within 5%),beloworabove. The shim always exits 0, so the eval runs after any outcome.skippedwith its reason: no prompts, a curve without acollectionblock, TileRT, or leader-only Dynamo vLLM DP metrics. Dynamo TRT-LLM and llm-d are gated: srtctl publishes TRT-LLM metrics by default, and a recipe that turns them off reportsno-counters.al-gate-prompts.ymlis now also callable:run-sweep.ymlande2e-tests.ymlcall it to hand the cached prompts to their agentic eval jobs. If it fails, the evals run without the gate and the sweep stays green.collect-evalsadds a gate table to its summary.collectionblocks. DeepSeek-V4.1-Flash was collected with vLLM'sdeepseek_v41tokenizer mode, soinfx/bench_serving/encoding_dsv41.pycopies vLLM's V4.1 encoder from the week of that collection (later vLLM renders reasoning efforthighas 75, not 50). Muse Spark was collected by SGLang'sbench_serving, which sends norepetition_penalty; the gate replays it on/v1/completions.Runtimeand itsbootstrap(), so a wrongbootstrapcall or a missingRuntime.aiperffails them.Removed: run-sweep's
al-gate-select,al-gate,al-gate-multinodeandal-gate-collectjobs; the templates'al-gateinput and gate-only steps; the launcher'sAL_GATE*request fields and validator,gate_args,check_gate;infx/workflows/al_gate_reuse.py; the gate-job filter incalc_success_rate.py.Coverage trade-off: only deployments that run agentic evals are gated; #4654 gated every speculative deployment and drafting configuration.
Not yet run on a cluster in this form. No config keys change, so there is no perf-changelog entry.
AI model disclosure
Runtimeregression tests intest_al_gate.py, adapted here.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
AgentX 吞吐量测试会把每个投机解码部署的接受长度强制设为黄金 AL，而评测无法反映草稿质量（投机解码是无损的），因此较弱的草稿模型不会被发现。#4654 在单独的门禁作业中测量真实 AL，每个作业各自启动服务。本 PR 把门禁移入现有的 AgentX 评测作业：门禁与评测共用同一次服务启动，并移除单独的门禁作业。门禁仍仅作报告，不会使评测失败或阻塞提交。
benchmarks/srt_al_gate.sh，再运行基准后评测。只有基准阶段能拿到各 worker 的指标端点。作业以EVAL_ONLY=false RUN_EVAL=true提交并关闭遥测；srt_eval.sh为评测恢复EVAL_ONLY=true。服务保持真实验证，与现有仅评测服务一致。infx.bench al-gate以并发 64 重放曲线记录的 SPEED-Bench 收集设置，从引擎自身 Prometheus 计数器计算AL = 1 + accepted / verifications，并写出al_gate_<RESULT_FILENAME>.json（pass为偏差 5% 以内，另有below/above）。脚本总是以 0 退出，评测随后照常运行。skipped记录原因：无提示词、曲线缺少collection块、TileRT，或仅暴露 leader 指标的 Dynamo vLLM DP。Dynamo TRT-LLM 与 llm-d 会执行门禁：srtctl 默认发布 TRT-LLM 指标，关闭指标的配方报告no-counters。al-gate-prompts.yml现在也可被调用：run-sweep.yml与e2e-tests.yml调用它把缓存的提示词交给 AgentX 评测作业；失败时评测不做门禁，扫描保持成功。collect-evals在摘要中加入门禁表。collection块。DeepSeek-V4.1-Flash 使用 vLLM 的deepseek_v41分词器模式采集，因此infx/bench_serving/encoding_dsv41.py复制了采集当周 vLLM 的 V4.1 编码器（之后的 vLLM 把推理强度high渲染为 75，而不是 50）。Muse Spark 由 SGLang 的bench_serving采集，不发送repetition_penalty；门禁在/v1/completions上重放。Runtime及其bootstrap()，错误的bootstrap调用或缺少Runtime.aiperf都会使测试失败。移除： run-sweep 的
al-gate-select、al-gate、al-gate-multinode、al-gate-collect作业；模板的al-gate输入及门禁专用步骤；启动器的AL_GATE*请求字段与校验、gate_args、check_gate；infx/workflows/al_gate_reuse.py；calc_success_rate.py中的门禁作业过滤。覆盖范围取舍： 只有运行 AgentX 评测的部署会执行门禁；#4654 会对每个投机解码部署及草稿配置执行门禁。
该形式的门禁尚未在集群上运行。 没有配置键变化，因此不新增 perf-changelog 条目。
AI 模型使用说明：claude-opus-5-5（Claude Code）在 #4654 的门禁代码上设计并实现评测内门禁，编写测试与文档，并撰写本说明；Devin AI（无法确认具体模型）编写了
test_al_gate.py中使用真实Runtime的回归测试，本 PR 对其做了改写。