Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
9b713e7
operatorx: dense GEMM serving benchmark with integrated profiling met…
hbarclay Sep 21, 2026
4a07695
operatorx: reduce to gemm-only baseline (NVIDIA + AMD)
hbarclay Sep 21, 2026
0b53a9b
operatorx: fix trace retention for OPERATORX_PROFILE_TRACE_EVERY=1
hbarclay Sep 21, 2026
91d52d4
operatorx: cold-cache profiling replays by default
hbarclay Sep 22, 2026
5159ce3
operatorx: per-op GPU telemetry with power-throttle retry
hbarclay Sep 22, 2026
1bf2752
operatorx: FIXME(Harrison) - migrate NVML telemetry to DCGM later
hbarclay Sep 22, 2026
227dc9c
operatorx: FIXME owner tag hbarclay
hbarclay Sep 22, 2026
8f7da23
operatorx: fix AMD backend import, telemetry windowing and device map…
hbarclay Sep 22, 2026
ae0785f
operatorx: record all NVML/amdsmi clocks, temperatures and throttle c…
hbarclay Sep 22, 2026
7bc02a6
operatorx: GEMM through vLLM's own linear layers with explicit quanti…
hbarclay Sep 23, 2026
88af2c2
operatorx: testlist generator takes GEMM dtypes from per-checkpoint r…
hbarclay Sep 23, 2026
8cabb91
operatorx: GEMM operands as vLLM-style quant descriptors
hbarclay Sep 23, 2026
f0eda07
operatorx: plain descriptions of the GEMM operand schema
hbarclay Sep 23, 2026
2322ee6
operatorx: move shared runner modules into runners/common
hbarclay Sep 23, 2026
c04e2e9
operatorx: drop scripts/ from the tree; run docs use python -m operatorx
hbarclay Sep 23, 2026
b5ea3dd
operatorx: restore upstream scripts/ and its docs
hbarclay Sep 23, 2026
7cd7866
operatorx: moe_layer op schema
hbarclay Sep 24, 2026
f16573f
operatorx: backend-chosen CUDA graphs, per-op kernel timeline, vLLM b…
hbarclay Sep 24, 2026
7265592
operatorx: gemm operand 'input' dtype for pre-quantized operands
hbarclay Sep 24, 2026
d5e4a5b
operatorx: GPU spin before each profiled replay; vLLM context builds …
hbarclay Sep 24, 2026
01efd18
operatorx: record CUDA-graph use as metrics.cuda_graph
hbarclay Sep 24, 2026
b1a1bb3
operatorx: ops/moe_layer.py -> ops/moe.py
hbarclay Sep 24, 2026
f5ee089
operatorx: drop the tests this PR added
hbarclay Sep 24, 2026
4496216
operatorx: make CI match the backends and testlists on this branch
hbarclay Sep 24, 2026
1a972e6
operatorx: moe_layer_small testlist (one full-size layer per Inferenc…
hbarclay Sep 24, 2026
3d3faf1
operatorx CI: counters mode (ncu / rocprofv3); launch ranks by path
hbarclay Sep 24, 2026
e8fcbd9
operatorx: moe_layer vLLM backend
hbarclay Sep 24, 2026
06d0388
operatorx: document CI counters mode and the moe_layer backend
hbarclay Sep 24, 2026
030a150
operatorx CI: MI300X Slurm partition is compute-0
hbarclay Sep 24, 2026
1068c77
operatorx CI: find the compute node's Nsight Compute for counters runs
hbarclay Sep 24, 2026
f4e7eba
operatorx: moe_layer latent experts, hash routing, DSv4 FP4 experts, …
hbarclay Sep 24, 2026
2e3265d
operatorx: free the previous MoE layer before building the next
hbarclay Sep 24, 2026
fcb9ab5
operatorx CI: mount the node's Nsight Compute tree at its own path
hbarclay Sep 24, 2026
432922b
operatorx CI: pick the newest real Nsight Compute install over a cuda…
hbarclay Sep 24, 2026
243b17e
operatorx: Qwen shared-expert gate applied inside the shared MLP, as …
hbarclay Sep 24, 2026
5ec27db
operatorx profiling: split replays at the spin kernel, not at any fil…
hbarclay Sep 24, 2026
ef8df9e
operatorx: runners/common/vllm/{linear,moe}.py; InferenceX recipe ima…
hbarclay Sep 24, 2026
38cf863
operatorx CI: fall back to the backend image when a recipe's image ta…
hbarclay Sep 24, 2026
60f5147
operatorx profiling: a platform-independent trace core with a per-pla…
hbarclay Sep 24, 2026
df91c1e
operatorx: rename op moe_layer -> moe
hbarclay Sep 25, 2026
ec4e092
operatorx: remove moe_gemm
hbarclay Sep 25, 2026
0effa38
operatorx: only gemm and moe - remove the other MoE variants
hbarclay Sep 25, 2026
dd147ff
operatorx CI: request runners by their cluster: label
hbarclay Sep 25, 2026
982201c
operatorx CI: runners are named by their cluster: label only
hbarclay Sep 25, 2026
4358817
operatorx: drop the InferenceX-matrix testlist generator
hbarclay Sep 25, 2026
e5050ff
operatorx: every case says where it comes from - sources and roles
hbarclay Sep 25, 2026
8b8353a
operatorx: widen the MoE list to 34 checkpoints and sweep the batch
hbarclay Sep 25, 2026
094d689
operatorx MoE: biased MXFP4 experts go to vLLM's own mxfp4 method
hbarclay Sep 26, 2026
cbbf581
operatorx: a gpt-oss-only MoE testlist
hbarclay Sep 26, 2026
f4bb356
operatorx: drop gpt-oss from the MoE lists for now
hbarclay Sep 26, 2026
39c81eb
operatorx sweep: dispatch the OperatorX database ingest when a run fi…
hbarclay Sep 26, 2026
ca97633
operatorx: NVIDIA and AMD only
hbarclay Sep 26, 2026
20f5df8
operatorx: validate every case against its op schema before it runs
hbarclay Sep 26, 2026
559e7d7
operatorx telemetry: stop the sampler even when the timed replay raises
hbarclay Sep 26, 2026
dbaaa0a
operatorx CI: resolve a recipe's fallback image digest once
hbarclay Sep 26, 2026
57a5941
operatorx moe: operands a1/w1/w2/a2 directly on experts and shared
hbarclay Sep 26, 2026
578ecbe
operatorx: sources are <checkpoint>/<role> pairs; name is gone
hbarclay Sep 26, 2026
2fcb42a
operatorx CI: import H100 DGXC images with the node's own enroot paths
hbarclay Sep 26, 2026
cebec1e
operatorx: DeepSeek-V4, Kimi-K3 and MiniMax-M3 gemm/moe testlists ove…
hbarclay Sep 27, 2026
b39833c
operatorx CI: ingest input; test runs dispatch with ingest=false
hbarclay Sep 27, 2026
d3defd5
operatorx: attention module op schemas (mla, mla_dsa, dsv4_attn, gqa,…
hbarclay Sep 27, 2026
0822910
operatorx: attention modules through vLLM's model runner (mla, gqa, gdn)
hbarclay Sep 27, 2026
649d780
operatorx: vLLM attention for Kimi-K3, GLM-5.x, Qwen3.8 and DeepSeek-…
hbarclay Sep 28, 2026
a5da603
operatorx: attention testlists for DeepSeek-V4, Kimi-K3 and MiniMax-M…
hbarclay Sep 28, 2026
e6cebad
operatorx attention: replay in inference mode, allocate like the sche…
hbarclay Sep 28, 2026
1ade4e4
operatorx CI: AMD pools also run the attention modules
hbarclay Sep 28, 2026
5f262ef
operatorx attention: replay under vLLM's current config too (FlashInf…
hbarclay Sep 28, 2026
96753c1
operatorx attention: schedule under vLLM's current config, as the wor…
hbarclay Sep 28, 2026
86b686c
operatorx: paths for the top-level layout; drop the removed generator…
hbarclay Sep 28, 2026
be67e62
operatorx attention: ask backends for CUDA-graph support under vLLM's…
hbarclay Sep 28, 2026
20a4bb0
operatorx attention: vLLM's default CUDA-graph capture ceiling, indep…
hbarclay Sep 28, 2026
e7ac530
Revert "operatorx attention: vLLM's default CUDA-graph capture ceilin…
hbarclay Sep 28, 2026
cd95f78
operatorx: cases run under InferenceX's srt-slurm recipes
hbarclay Sep 28, 2026
55a42ed
operatorx: multi-device gemm/moe (tp/dp/ep) under InferenceX recipes …
hbarclay Sep 28, 2026
7945da6
operatorx CI: plan with a backend image's recorded digest once the re…
hbarclay Sep 30, 2026
d71caa9
operatorx: main's changes since the last rebase
hbarclay Oct 22, 2026
4652976
operatorx: find recipes through configs/*-master.yaml (srt-recipe-dir…
hbarclay Oct 22, 2026
b8e79c9
operatorx CI: record the AMD vLLM image's digest; its nightly tag is …
hbarclay Oct 22, 2026
7efe1b2
operatorx: run on InferenceX's images; digest-pinned images import by…
hbarclay Oct 22, 2026
739802c
operatorx CI: CollectiveX's current AMD partitions; load ep_harness w…
hbarclay Oct 22, 2026
c48a582
operatorx: cross-rank iteration reduction through CollectiveX's _Coll…
hbarclay Oct 22, 2026
13751be
operatorx CI: fall back to the backend's image for any pruned Inferen…
hbarclay Oct 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 66 additions & 19 deletions .github/workflows/operatorx-sweep.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,49 @@ name: OperatorX Sweep
on:
workflow_dispatch:
inputs:
pool:
description: GPU runner pool (one physical node per shard)
runner:
description: Self-hosted runner label (one physical node per shard)
type: choice
default: h100-dgxc
options: [h100-dgxc, h200-dgxc, b200-nscale, b300, gb200, gb300, mi300x, mi325x, mi355x]
default: cluster:h100-dsxe
options:
- cluster:h100-dsxe
- cluster:h200-dgxc
- cluster:b200-nscale
- cluster:b300-dsxe
- cluster:gb200-nv
- cluster:gb300-nv
- cluster:mi300x-amd
- cluster:mi325x-amds
- cluster:mi355x-amds
backends:
description: Comma-separated backends (torch GEMM/attention, AMD aiter attention, vllm routed MoE)
description: Comma-separated backends (vllm GEMM; AMD also torch GEMM)
type: string
default: torch
default: vllm
testlists:
description: Comma-separated testlist stems
type: string
default: gemm
mode:
description: timing (latency, telemetry, profiler replay) or counters (per-kernel ncu / rocprofv3 counters; latencies perturbed)
type: choice
default: timing
options: [timing, counters]
world_sizes:
description: Comma-separated world sizes (1,2,4; also 8 on eight-GPU pools)
description: Comma-separated world sizes (1,2,4; also 8 on eight-GPU nodes)
type: string
default: '1'
chunk_size:
description: Maximum shapes per shard (1-500)
type: number
default: 500
recovery_run_id:
description: Optional recent OperatorX run on this pool whose failed cleanup should be retried
description: Optional recent OperatorX run on this runner whose failed cleanup should be retried
type: string
default: ''
ingest:
description: Ingest the results into the OperatorX database (false for test runs)
type: boolean
default: true
pull_request:
paths:
- operatorx/**
Expand Down Expand Up @@ -60,22 +78,27 @@ jobs:
- name: Plan requested coverage
id: plan
env:
OPERATORX_POOL: ${{ inputs.pool || 'h100-dgxc' }}
OPERATORX_BACKENDS: ${{ inputs.backends || 'torch' }}
OPERATORX_RUNNER: ${{ inputs.runner || 'cluster:h100-dsxe' }}
OPERATORX_BACKENDS: ${{ inputs.backends || 'vllm' }}
OPERATORX_TESTLISTS: ${{ inputs.testlists || 'gemm' }}
OPERATORX_WORLD_SIZES: ${{ inputs.world_sizes || '1' }}
OPERATORX_MODE: ${{ inputs.mode || 'timing' }}
OPERATORX_RECOVERY_RUN: ${{ inputs.recovery_run_id }}
OPERATORX_CHUNK_SIZE: ${{ github.event_name == 'workflow_dispatch' && format('{0}', inputs.chunk_size) || '500' }}
PYTHONPATH: .:inferencex-e2e
run: |
set -eo pipefail
source inferencex-e2e/benchmarks/check_env.sh
check_env_vars OPERATORX_POOL OPERATORX_BACKENDS OPERATORX_TESTLISTS OPERATORX_WORLD_SIZES OPERATORX_CHUNK_SIZE GITHUB_RUN_ID GITHUB_RUN_ATTEMPT GITHUB_SHA GITHUB_OUTPUT
python3 -m operatorx.ci plan \
check_env_vars OPERATORX_RUNNER OPERATORX_BACKENDS OPERATORX_TESTLISTS OPERATORX_WORLD_SIZES OPERATORX_MODE OPERATORX_CHUNK_SIZE GITHUB_RUN_ID GITHUB_RUN_ATTEMPT GITHUB_SHA GITHUB_OUTPUT
# Cases take InferenceX's srt-slurm recipes, expanded by InferenceX's own code (srtctl).
git submodule update --init inferencex-e2e/utils/srt-slurm
uv run --no-project --exclude-newer PT12H --python 3.12 --with pyyaml --with ruamel.yaml \
--with marshmallow --with marshmallow-dataclass --with pydantic --with jinja2 --with requests --with rich \
python -m operatorx.ci plan \
--platform-config operatorx/platforms.json \
--pool "$OPERATORX_POOL" --backends "$OPERATORX_BACKENDS" \
--runner "$OPERATORX_RUNNER" --backends "$OPERATORX_BACKENDS" \
--testlists "$OPERATORX_TESTLISTS" --world-sizes "$OPERATORX_WORLD_SIZES" \
--chunk-size "$OPERATORX_CHUNK_SIZE" --run-id "$GITHUB_RUN_ID" \
--chunk-size "$OPERATORX_CHUNK_SIZE" --mode "$OPERATORX_MODE" --run-id "$GITHUB_RUN_ID" \
--attempt "$GITHUB_RUN_ATTEMPT" --source-sha "$GITHUB_SHA" \
--out operatorx-manifest.json > operatorx-matrix.json
if [ -n "$OPERATORX_RECOVERY_RUN" ]; then
Expand Down Expand Up @@ -106,10 +129,10 @@ jobs:
runs-on: >- # zizmor: ignore[self-hosted-runner] GPU Slurm runners by design; this job runs only on workflow_dispatch
${{ fromJSON(vars.PRIORITY_SCHEDULER_ENABLED == 'true' &&
format('["self-hosted",{0},"nodes:1",{1},{2}]',
toJSON(matrix.pool),
toJSON(matrix.runner),
toJSON(format('ci-job-{0}-{1}', needs.setup.outputs.priority, matrix.queue-token)),
toJSON(format('ci-attempt-{0}', github.run_attempt))) ||
format('["self-hosted",{0}]', toJSON(matrix.pool))) }}
format('["self-hosted",{0}]', toJSON(matrix.runner))) }}
name: p${{ needs.setup.outputs.priority }} | OperatorX ${{ matrix.id }}
env:
OPERATORX_SHARD: ${{ matrix.id }}
Expand Down Expand Up @@ -149,13 +172,13 @@ jobs:
if: inputs.recovery_run_id != ''
env:
OPERATORX_RECOVERY_RUN: ${{ inputs.recovery_run_id }}
OPERATORX_POOL: ${{ matrix.pool }}
OPERATORX_RUNNER: ${{ matrix.runner }}
run: |
set -eo pipefail
source inferencex-e2e/benchmarks/check_env.sh
check_env_vars RUNNER_TEMP OPERATORX_RECOVERY_RUN OPERATORX_POOL OPERATORX_CLEANUP_SECONDS
check_env_vars RUNNER_TEMP OPERATORX_RECOVERY_RUN OPERATORX_RUNNER OPERATORX_CLEANUP_SECONDS
python3 -m operatorx.ci recover --artifacts operatorx-recovery \
--run-id "$OPERATORX_RECOVERY_RUN" --pool "$OPERATORX_POOL" \
--run-id "$OPERATORX_RECOVERY_RUN" --runner "$OPERATORX_RUNNER" \
--cleanup-seconds "$OPERATORX_CLEANUP_SECONDS" \
--platform-config operatorx/platforms.json
- name: Run one Slurm allocation
Expand Down Expand Up @@ -233,3 +256,27 @@ jobs:
path: operatorx-summary.json
if-no-files-found: error
retention-days: 14

trigger-ingest:
name: Trigger OperatorX database ingest
# A run with failed shards is still stored: its unrun cases read as missing.
if: >-
always() && github.event_name == 'workflow_dispatch' && inputs.ingest &&
(needs.summarize.result == 'success' || needs.summarize.result == 'failure')
needs: [summarize]
runs-on: ubuntu-latest
steps:
- name: Dispatch ingest to InferenceX-app
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
RUN_ID: ${{ github.run_id }}
with:
# Cross-repository dispatch credential, as run-sweep.yml's database ingest uses.
github-token: ${{ secrets.FRONTEND_PAT }} # zizmor: ignore[secrets-outside-env]
script: |
await github.rest.repos.createDispatchEvent({
owner: "SemiAnalysisAI",
repo: "InferenceX-app",
event_type: "ingest-operatorx",
client_payload: { "run-id": process.env.RUN_ID }
});
Loading
Loading