Repository navigation
fix(power): sample power-telemetry DCGM every 1000 ms / fix(power)：功耗遥测 DCGM 改为每 1000 ms 采样 - #4974
Merged
Merged
Conversation
The power-telemetry dcgm-exporter sampled every 100 ms, ten times as often as the collector scrapes it. On GB300 disaggregated serving that interval slowed prefill-to-decode KV transfer: with identical recipes, the PR 4747 run (Tachometer's 1000 ms exporter only) moved KV at 118.5 GB/s during a c176 request burst against 52.6 GB/s for the PR 4834 run (100 ms power exporter), and e2e p90 rose from 3.67 to 4.33 s. Sample every 1000 ms, matching the 1 s scrape and each cluster's default GPU exporter, and append a perf-changelog entry for the 16 power: true AgentX configs whose fingerprint includes the telemetry block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
移除功耗遥测的专项扫描记录，让相关配置在下次扫描时采用 1000 ms 采样间隔。
edwingao28
approved these changes
Nov 11, 2026
richardhuo-nv
added a commit
that referenced
this pull request
Nov 11, 2026
Power telemetry now samples DCGM every 1000 ms (#4974), so run the 1P2D and 2P2D points at their original concurrency again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adibarra
added a commit
that referenced
this pull request
Nov 12, 2026
…ew points / perf(qwen3.5)：GB300 TRT AgentX 升级至 TRT-LLM 1.4.0rc3 并更新为十个新配置点 (#4834) * perf(qwen3.5): update GB300 TRT AgentX to TRT-LLM 1.4.0rc3 with a new ten-point search space Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf-changelog: set PR link Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf-changelog: restore existing entries; set PR link on the new entry only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): set GB300 TRT AgentX's TRT-LLM 1.4.0rc3 options as config fields TensorRT-LLM 1.4.0rc3 (NVIDIA/TensorRT-LLM#20815) reads none of the feature-branch TRTLLM_* switches this recipe set. Its Qwen3.5 paths are on by default, derived from existing config, or explicit LLM-API fields. - Prefill: shortest-uncached-first context scheduling, breakable prefill CUDA graphs on the same buckets without torch.compile, and branch snapshots under block_reuse_config. DEP prefill keeps piecewise graphs and ages context requests after 30 scheduler iterations (about 8 s at its measured 3.8 iterations/s). - Decode: a replicated embedding table. TEP decode drafts with the MXFP8 MTP LM head and keeps the GEN KV-transfer window with an explicit max_tokens_in_buffer; 3P1D also sends the MNNVL all-reduce payload in E4M3. - Drop the switches rc3 does not read: TRTLLM_Q35_*, SRPT, PREFILL_BCG, in-forward snapshot, disaggregated-GEN, spec-sampler, overlap-loop, MNNVL FP8 and cuBLAS FP8 tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): drop the MXFP8 MTP draft LM head from GB300 TRT AgentX Check 13 requires the draft to run as the checkpoint ships it. The Qwen3.5-397B-A17B-NVFP4-V2 LM head is BF16, and draft_lm_head_dtype: mxfp8 sampled the MTP drafts from an MXFP8 copy of it; TEP decode now drafts from the BF16 head, the rc3 default. The c720 override no longer needs to remove the field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): update GB300 TRT AgentX to TRT-LLM 1.4.0rc3 with ten new points Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): lower GB300 TRT AgentX conc 128 and 176 points by 10% Run the 1P2D TEP4/TEP8 point at conc 115 instead of 128 and the 2P2D TEP2/TEP8 point at conc 158 instead of 176, and rename their overrides to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): restore GB300 TRT AgentX conc 128 and 176 points Power telemetry now samples DCGM every 1000 ms (#4974), so run the 1P2D and 2P2D points at their original concurrency again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Since #3670, every multi-node point on a cluster with a
power-exporter-portstarts the power-telemetry dcgm-exporter fromconfigs/srt-recipes/telemetry-dcgm.yaml, and that exporter sampled every 100 ms (--collect-interval=100). When power telemetry brings its own exporter, srt-slurm skips Tachometer's (implicit.py#L168-L171), so it is the only DCGM exporter on each node. The collector reads it once per second (collect_interval_ms: 1000), so most of those samples are never read, but the sampling still slows serving. srt-slurm measured about 2% decode ITL p50 overhead for 100 ms sampling on GB300 (telemetry_stage.py#L50-L53). Its warning only covers Tachometer's exporter, though, so these runs never logged it.This PR samples every 1000 ms (
--collect-interval=1000). That matches the 1 s scrape and thedefault_gpu_exporterof every NVIDIA cluster inconfigs/runners.yaml, which is what runs sampled at before power telemetry was on by default.Evidence: PR 4747 vs PR 4834
PRs #4747 and #4834 ran the same GB300 Qwen3.5 TRT-LLM AgentX curve. The PR 4747 run predates #3670: it had no power telemetry, only Tachometer's 1000 ms exporter. The PR 4834 runs had the 100 ms power exporter. The regression was isolated to this sampling interval. The other differences between the runs were ruled out: srt-slurm 2.66.1 → 2.66.2, agentx-harness 1.0.12 → 1.0.13, and node placement.
e2e p90 is higher at every point, by 5–18% on the eight points whose recipe fingerprints match. ITL p50 rises by about 2% on average, in line with srt-slurm's measurement, and throughput per GPU stays within 1.4%.
¹ c8 and c28 failed at setup in run 42317315774, so their 4834 numbers come from the PR's next sweep, run 42390081277, with the same recipes.
² 4834 drops
mnnvl_allreduce_fp8_payload: truefrom the 3P1D decode used at c352 and c392.³ 4747's c720 ran twice; this is the last attempt.
At c128 and c176, which were analyzed in detail, most of the gap comes from one request burst about 300 s into the run. At c176, prefill → decode KV transfer during the burst (300–360 s) moved 7,111 GB at 118.5 GB/s in the 4747 run against 3,154 GB at 52.6 GB/s in the 4834 run, and the backed-up requests landed in the latency tail. Leaving out 300–600 s of each run, e2e p90 barely moves: c128 goes from 2.90 to 2.92 s and c176 from 3.39 to 3.41 s.
sweep_*.logstartsdcgm-exporter --collect-interval=1000 --address :9402on every node.power/manifest.jsonrecordsdcgm-exporter --collect-interval=100 --address :19401.Sweep impact
The changelog entry lists 19 config keys, 107 AgentX points on H200, B200, B300, GB200 and GB300, so this PR needs a green sweep and
/usebefore merge. No sweep label is applied yet.infx.matrix.changedlists: the rows withpower: true, whose recipe fingerprint includes the telemetry block.power, but their published results came from sweeps that already included [PowerX] collect multinode power by default / [PowerX] 默认采集多节点功耗 #3670, so they ran the 100 ms exporter. These areqwen3.5-fp8-b300-dynamo-sglang-agentic-disagg(reused run 42278372774 from [PowerX] collect Qwen3.5 FP8 B300 Dynamo-SGLang AgentX power / [PowerX] 采集 Qwen3.5 FP8 B300 Dynamo-SGLang AgentX 功耗 #4413), andkimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disaggandkimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk(reused run 42376772667 from [NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: retune disaggregated and megakernel points / [NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang：重新调优分离式与 megakernel 配置点 #4893).For power data, each 1 s scrape now returns a sample up to 1 s old instead of up to 100 ms old. The number of samples per window is unchanged.
Six Qwen3.5 SGLang GB200/GB300 1k1k recipes still set their own 100 ms telemetry command. No master config uses them, so they are left as is.
AI model disclosure
claude-opus-5-5), via Claude CodePR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概述
自 #3670 起，在配置了
power-exporter-port的集群上，每个多节点点位都会按configs/srt-recipes/telemetry-dcgm.yaml启动功耗遥测的 dcgm-exporter，它的采样间隔是 100 ms（--collect-interval=100）。功耗遥测自带 exporter 时，srt-slurm 会跳过 Tachometer 的 exporter（implicit.py#L168-L171），所以它是每个节点上唯一的 DCGM exporter。采集器每秒只读取一次（collect_interval_ms: 1000），大部分样本根本不会被读到，但采样本身仍然拖慢推理。srt-slurm 自己测得，GB300 上 100 ms 采样会让解码 ITL p50 增加约 2%（telemetry_stage.py#L50-L53）。不过这条告警只针对 Tachometer 的 exporter，所以这些运行里从未输出过。本 PR 将采样间隔改为 1000 ms（
--collect-interval=1000）。这与 1 秒的采集周期一致，也与configs/runners.yaml中所有 NVIDIA 集群的default_gpu_exporter一致，即功耗遥测默认开启之前的采样间隔。证据：PR 4747 与 PR 4834
#4747 和 #4834 运行的是同一条 GB300 Qwen3.5 TRT-LLM AgentX 曲线。PR 4747 的运行早于 #3670：没有功耗遥测，只有 Tachometer 的 1000 ms exporter。PR 4834 的运行使用了 100 ms 的功耗 exporter。性能回退已定位到这一采样间隔。几次运行之间的其他差异均已排除：srt-slurm 2.66.1 → 2.66.2、agentx-harness 1.0.12 → 1.0.13，以及节点分配。
所有点位的 e2e p90 都变高了；在 recipe 指纹相同的八个点位上，升幅为 5–18%。ITL p50 平均上升约 2%，与 srt-slurm 的测量结果一致；单 GPU 吞吐量的变化不超过 1.4%。
表格见上方英文部分。
¹ c8 和 c28 在 run 42317315774 中启动阶段就失败了，因此它们的 4834 数据取自该 PR 的下一次扫描 run 42390081277，recipe 相同。
² 4834 去掉了 c352 和 c392 所用 3P1D 解码配置中的
mnnvl_allreduce_fp8_payload: true。³ 4747 的 c720 运行了两次，这里取最后一次。
在详细分析过的 c128 和 c176 上，差距主要来自运行约 300 s 时的一次突发请求。在 c176 上，突发期间（300–360 s）预填充 → 解码的 KV 传输，4747 的运行以 118.5 GB/s 传输了 7,111 GB，4834 的运行则以 52.6 GB/s 只传输了 3,154 GB，积压的请求落入了延迟尾部。去掉每次运行的 300–600 s 后，e2e p90 几乎不变：c128 从 2.90 s 变为 2.92 s，c176 从 3.39 s 变为 3.41 s。
各次运行和服务器日志产物的链接见上方英文部分。
扫描影响
changelog 记录列出 19 个配置键，共 107 个 AgentX 点位，分布在 H200、B200、B300、GB200 和 GB300 上，因此合并前需要一次全绿的扫描和
/use。目前尚未添加扫描标签。infx.matrix.changed列出的配置键，即设置了power: true的行，它们的 recipe 指纹包含遥测配置块。power，但它们已发布的结果来自已包含 [PowerX] collect multinode power by default / [PowerX] 默认采集多节点功耗 #3670 的扫描，运行时用的就是 100 ms 的 exporter。它们是qwen3.5-fp8-b300-dynamo-sglang-agentic-disagg（[PowerX] collect Qwen3.5 FP8 B300 Dynamo-SGLang AgentX power / [PowerX] 采集 Qwen3.5 FP8 B300 Dynamo-SGLang AgentX 功耗 #4413 复用的 run 42278372774），以及kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg和kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk（[NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: retune disaggregated and megakernel points / [NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang：重新调优分离式与 megakernel 配置点 #4893 复用的 run 42376772667）。功耗数据方面，每次 1 秒的采集读到的样本最多是 1 秒前的，而不是 100 ms 前的。每个窗口的样本数不变。
有六个 Qwen3.5 SGLang GB200/GB300 1k1k recipe 仍自带 100 ms 的遥测命令。没有任何主配置使用它们，因此保持不变。
AI 模型使用说明：Claude Opus 5.5（
claude-opus-5-5），通过 Claude Code 使用。它对比了 PR 4747 与 PR 4834 的运行，梳理了 srt-slurm 中 DCGM exporter 的启动路径，完成了配置修改和 changelog 记录，并撰写了本说明。性能回退由作者定位到采样间隔。未使用其他模型或委派 agent。🤖 Generated with Claude Code