Skip to content

[PowerX] collect AMD multinode power with the native exporter / [PowerX] 用原生 exporter 采集 AMD 多节点功耗 - #4820

Merged
edwingao28 merged 3 commits into
feat/nvl72-power-defaultsfrom
feat/amd-multinode-power
Nov 5, 2026
Merged

edwingao28 merged 3 commits into
feat/nvl72-power-defaultsfrom
feat/amd-multinode-power

Conversation

@edwingao28

@edwingao28 edwingao28 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Multinode points on clusters without a DCGM exporter port now measure GPU power with the cluster's own exporter. Only mi355x-amds changes: its disaggregated and TileRT points get best-effort AMD GPU and RAPL CPU power. runners.yaml now rejects RAPL on shared nodes. No config key changes, so no changelog entry.

Testing: Unit tests pass; 19 of 1388 rendered recipes change, all mi355x multinode. MI355X proof has valid GPU and CPU power. TP4-prefill variants idle some GPUs, so their power is invalid; results stay.

Depends on: #3670.

AI model disclosure
  • Model/version: claude-opus-5-5 (Claude Opus 5.5)
  • Role: implementation, tests, docs, PR text
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

没有 DCGM exporter 端口的集群,多节点点现在用集群自带的 exporter 采集 GPU 功耗。目前只有 mi355x-amds 受影响:它的 disagg 和 TileRT 点默认采集 AMD GPU 与 RAPL CPU 功耗(best-effort)。runners.yaml 现在拒绝在共享节点上配置 RAPL。没有配置 key 变化,因此不新增 changelog 条目。

测试: 单元测试通过;1388 个渲染出的 recipe 中只有 19 个 mi355x 多节点 recipe 变化。MI355X 验证 run 的 GPU 功耗和 RAPL CPU 功耗均有效。TP4 prefill 变体有空闲 GPU,功耗会判为无效,但基准测试结果保留。

依赖: #3670。

AI 模型:claude-opus-5-5(Claude Opus 5.5),负责实现、测试、文档和 PR 文字。

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +143 -99

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@edwingao28
edwingao28 merged commit 93e5203 into feat/nvl72-power-defaults Nov 5, 2026
15 checks passed
@edwingao28
edwingao28 deleted the feat/amd-multinode-power branch November 5, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant