You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1e9feb6
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: .github/AGENT_OPERATIONS.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -74,15 +74,15 @@ Multinode srt-slurm results, and single-node fixed-sequence and AgentX results w
74
74
75
75
Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count.
76
76
77
-
A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for those jobs only, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor.
77
+
A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg; on `gb200-nv` and `gb300-nv` every multinode benchmark point binds the leg, and `cpu-power: true` makes it expected. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for jobs that bind it, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor.
78
78
79
79
The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU<n>:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. A valid GPU leg also publishes `workers[]`: one entry per worker per host (`role`, the manifest's `worker_idx`, `hosts` holding that one host, `num_gpus`, `avg_power_w`), so a worker spanning trays repeats its index once per host and can be modeled per tray; it is omitted when a GPU belongs to two workers.
80
80
81
-
The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.
81
+
The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or on AgentX lanes whose launcher validates power, except NVL72 points whose row omits `power`) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.
82
82
83
83
Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
84
84
85
-
A multinode master row with `power: true` enables official energy collection through the shared telemetry fragment; native single-node throughput and AgentX jobs enable telemetry through their bound recipe. Native single-node jobs inherit the exporter image, labels and metrics from the cluster's `default_gpu_exporter` in `inferencex-e2e/configs/runners.yaml`: DCGM on NVIDIA, the `kind: custom` AMD device-metrics-exporter on AMD. Multi-node power rows keep their shared DCGM telemetry block. Eval-only jobs omit collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated.
85
+
A multinode master row with `power: true` enables official energy collection through the shared telemetry fragment. On `gb200-nv` and `gb300-nv` every multinode benchmark point measures GPU and Grace CPU power whether or not its row sets `power`; invalid telemetry then records `power_valid: 0` and keeps the result, while `power: true` keeps invalid AgentX power fatal. Native single-node throughput and AgentX jobs enable telemetry through their bound recipe. Native single-node jobs inherit the exporter image, labels and metrics from the cluster's `default_gpu_exporter` in `inferencex-e2e/configs/runners.yaml`: DCGM on NVIDIA, the `kind: custom` AMD device-metrics-exporter on AMD. Multi-node power rows keep their shared DCGM telemetry block. Eval-only jobs omit collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated.
86
86
87
87
Power audit artifacts are named `power_audit_<result>` and contain the `power_validation*.json` sidecars plus, for native telemetry jobs, `LOGS/power/` (`samples.csv` v3 with `temperature_c`, `manifest.json`, `windows/`). They are uploaded even when validation fails.
Copy file name to clipboardExpand all lines: inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,9 +53,9 @@ All referenced recipes must be checked in: srt-slurm 2 ships curated examples in
53
53
54
54
Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments: native srt-slurm YAML holding only recipe-specific settings. At launch the fragment is composed and bound:
55
55
56
-
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
56
+
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). On `gb200-nv` and `gb300-nv` every multi-node benchmark point gets it with the Grace CPU exporter, unless its recipe runs etcd and NATS on a dedicated node. The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
57
57
2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning.
58
-
3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:<model>`, `model.container: <image>`, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments.
58
+
3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:<model>`, `model.container: <image>`, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power, or any GB200 or GB300 AgentX point the launcher measures by default) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments.
59
59
4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/<script>`, is one of the installers in the binder's `INSTALLERS`, the binder writes the point's `router` or `kv-offload-backend` version as `ROUTER_VERSION` or `KV_OFFLOAD_BACKEND_VERSION` into the top-level `environment` (workers and static routers) or into that service's `env` (services do not inherit `environment`). A point that does not declare the version fails, and a `vllm-router==` pin in `SETUP_PIP_PACKAGES` must equal the master router version.
60
60
61
61
A fragment that sets a key the binder or launcher writes, or that the workflow exports from the matrix point (`_POINT_KEYS`, `BOUND_ENV` in `workload.py`, and `ROUTER_VERSION` or `KV_OFFLOAD_BACKEND_VERSION` in any env; single-node variants may still name `CONC`), fails before submission, even with the bound value. Values `benchmarks/runtime_settings.sh` and the multi-node `FRAMEWORK` settings export reach every process through the job environment, so a fragment sets one only to override it (an AgentX client `TOKENIZER` names a local checkpoint while `MODEL` stays the published Hugging Face id). `hf:<model>` resolves to the cluster's staged checkpoint (`models.entries`) unless a `models.OVERRIDES` row serves the Hub snapshot, and the master image to its staged container. A multi-node fixed-sequence point with neither checkpoint nor Hub row fails before submission.
@@ -72,7 +72,7 @@ It writes one bound recipe per fixed-sequence or AgentX point, validated by the
72
72
73
73
## Recipe fingerprints
74
74
75
-
The planner gives every benchmark row a `recipe-fingerprint`: a SHA-256 of its matrix fields (except `conc` and `exp-name`) and the bound variant the launcher submits, composed with its shared block and the native single-node or multi-node `power: true` telemetry block. Concurrency values, the job `name` and everything the launcher adds for a cluster stay out, so a recipe keeps one fingerprint across concurrencies and clusters. An `eval-srt-recipe` contributes only its path; rows without an srt-slurm recipe hash their matrix fields alone.
75
+
The planner gives every benchmark row a `recipe-fingerprint`: a SHA-256 of its matrix fields (except `conc` and `exp-name`) and the bound variant the launcher submits, composed with its shared block and the native single-node or multi-node `power: true` telemetry block; the GB200 and GB300 default telemetry stays out. Concurrency values, the job `name` and everything the launcher adds for a cluster stay out, so a recipe keeps one fingerprint across concurrencies and clusters. An `eval-srt-recipe` contributes only its path; rows without an srt-slurm recipe hash their matrix fields alone.
76
76
77
77
The planner selects variants without srtctl ([`variants.py`](../../../infx/srt_slurm/variants.py)), the way the launcher does, so a row that no variant serves fails planning instead of its launch. Before writing a `perf-changelog.yaml` entry, list the config keys whose results the change invalidates:
Copy file name to clipboardExpand all lines: inferencex-e2e/configs/CONFIGS.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -153,7 +153,7 @@ input.
153
153
154
154
Notes:
155
155
- The fields above are the common ones, not the full schema. The Pydantic models in [`infx/matrix/validation.py`](../infx/matrix/validation.py) are the authoritative contract; they also accept fields such as `spec-decoding`, `num-nodes`, `require-power` and `power`, and they reject any field they do not define, which fails matrix generation.
156
-
- A multinode search-space entry with `power: true` measures DCGM power: its rows carry `power: true`, the launcher merges the shared telemetry block into the recipe and rejects lanes without a power rule.
156
+
- A multinode search-space entry with `power: true` measures DCGM power: its rows carry `power: true`, the launcher merges the shared telemetry block into the recipe and rejects lanes without a power rule. On `gb200-nv` and `gb300-nv` every multinode benchmark point measures GPU and Grace CPU power without it, and keeps its result when telemetry fails; there `power: true` makes invalid AgentX power fail the point.
157
157
- A multinode entry with `cpu-power: true` beside `power: true` also measures CPU power: its rows carry `cpu-power: true` and the launcher binds the cluster's `cpu-power-exporter` record beside the DCGM block. Single-node entries cannot carry it.
158
158
- srt-slurm entries require `srt-recipe-dir`. Every row requires `srt-recipe: <file>[:<selector>]` under it (selector: `base`, `override_<name>` or `zip_override_<name>[<i>]`). Multinode rows may add `eval-srt-recipe`. Validation rejects missing files or variants, and `CONFIG_FILE`/`EVAL_CONFIG_FILE` settings. Only `llmd-vllm` keeps `CONFIG_FILE`. Generated rows carry the path from `inferencex-e2e/`.
159
159
- Setting the fields above only guarantees that their values are passed as environment variables to benchmark scripts. Single-node jobs receive `PP_SIZE`, `DCP_SIZE`, and `PCP_SIZE`. Multinode jobs receive `PREFILL_PP_SIZE`, `PREFILL_DCP_SIZE`, `PREFILL_PCP_SIZE`, `DECODE_PP_SIZE`, `DECODE_DCP_SIZE`, and `DECODE_PCP_SIZE`. Actually using those variables is an implementation detail of the benchmark Bash script.
0 commit comments