[hbarclay]

OperatorX measures single ops (a GEMM, one MoE layer) on NVIDIA and AMD. It runs them through the same framework layers InferenceX serves with, so the kernel it times is the kernel real inference uses. This PR replaces the prototype under experimental/operatorx/ with that version, adds the collection methodology, and connects the sweep to the OperatorX database.

Scope

• Ops: only gemm and moe for now.

  • Attention, collectives, moe_gemm, moe_forward, moe_layer, the other NVIDIA backends, the TPU and Trainium runners, and the scripts/inferencex_testlist generator are removed. Ops and platforms will come back one at a time.

  • gemm (ops/gemm.py): activation a [M, K] and weight b [N, K]. Each is an operand descriptor {"dtype", "scale"?, "scale2"?, "symmetric"?}.

    • scale is {"dtype", "static", "group": [rows, cols]}, and -1 spans a dimension.
    • So [-1,-1] is per-tensor, [1,-1] per-token, [-1,1] per-channel, [1,g] groups of g along K, and [r,c] 2-D blocks.

  • moe (ops/moe.py): one MoE layer from the router GEMM to the combined output. It has:

    • tokens and hidden;
    • experts: count, top-k, intermediate size, optional bias / latent / zero experts, and the operand descriptors a1 (input), w1 (fused gate/up weight), w2 (down weight) and a2 (intermediate activation), named after vLLM's FusedMoEQuantConfig slots;
    • router: gate dtype, scoring, selection, bias, renormalize, scale;
    • activation, optional shared experts, and a routing distribution (natural / balanced / zipf / single_hot).

    Kernels, dispatch, fusion and streams are the backend's choice.

  • Every case says where it comes from: sources lists one <checkpoint>/<role> per layer that runs it, e.g. deepseek-ai/DeepSeek-V4-Pro/attn.wq_b or zai-org/GLM-5-FP8/mlp.

    • A shape shared by several models, or by several layers of one model, lists every pair, so each role stays tied to its model. The role is what follows the last /.
    • DeepSeek-V4 roles come from the checkpoints' own weight maps under vLLM's TP sharding and fusions. That covers the indexer, compressor, router, mHC and head (lm_head) shapes.
    • It's metadata and doesn't change what's measured.

• Backends:

  • vllm gemm (NVIDIA, AMD; runners/common/vllm/linear.py).
    • Builds vLLM's ReplicatedLinear under the quant config the descriptors imply, loads synthetic checkpoint-format weights, and runs process_weights_after_loading. vLLM then does its own activation quantization, repacking and kernel selection.
    • Emulation-only paths are reported unsupported. So is a quantized layer that comes out unquantized.
  • vllm moe (runners/common/vllm/moe.py): vLLM's GateLinear router, FusedMoEFactory experts and shared-expert MLP, wired as vLLM's model blocks wire them. Covers DeepSeek-V4 FP4 experts and hash routing, Kimi-K3 latent experts, Qwen's shared-expert gate, and forced expert loads.
  • torch (AMD): hipBLASLt bf16 and per-tensor fp8.

• Testlists:

  Testlist	Cases	What
  gemm.json	5,416	InferenceX matrix shapes under each checkpoint's per-layer quant scheme
  gemm_serving_{8k1k,all}_min.json	1,505 / 8,099	DeepSeek-V4-architecture shapes; sources is DSv4-Pro where the shape is one of its projections, else []
  gemm_perf.json	11	BF16 smoke list
  moe_small.json	62	One full-size MoE layer per InferenceX MoE checkpoint scheme (32 checkpoints) at tokens=1; one layer per family swept at 8/64/512/2048 tokens; DeepSeek-R1 / Qwen3.5 FP8 under each expert-load distribution

  gpt-oss is left out of moe_small for now. Its biased MXFP4 experts don't go through vLLM's mxfp4 method cleanly here yet.

Measurement

• GPU timing:
  • Warmup has a wall-clock floor.
  • A GPU spin runs ahead of the timed loop, so launch overhead is excluded.
  • The cache is flushed before every iteration; the result is the median of 10.
• Telemetry (runners/common/telemetry.py):
  • NVML / amdsmi record clocks, power, temperatures and throttle reasons while each op runs.
  • Capped attempts are retried with growing inter-kernel sleeps.
• Profiling (runners/common/profiling.py, runners/common/trace.py):
  • A torch.profiler replay attaches a per-kernel breakdown (metrics.profile).
  • A platform-independent trace module (trace.py) with a small adapter per platform (CUDA/ROCm today).
• Counters mode: one marked replay per op under Nsight Compute or rocprofv3. Counter files go under results/counters/, and latencies are perturbed.

CI (operatorx-sweep.yml)

• Inputs:
  • runner: a cluster: label (cluster:h100-dgxc default, then h200-dgxc, b200-nscale, b300-dsxe, gb200-nv, gb300-nv, mi300x-amd, mi325x-amds, mi355x-amds);
  • backends, testlists, mode (timing | counters), world_sizes, chunk_size.
• New last job trigger-ingest: sends InferenceX-app an ingest-operatorx repository_dispatch with the run id, as run-sweep.yml does for benchmark results. The app stores the run in the OperatorX database (SemiAnalysisAI/InferenceX-app, branch hbarclay/operatorx). A run with failed shards is stored too; its unrun cases read as missing.
• Dispatched from hbarclay/operatorx:
  • gemm timing, 5,416 cases each: H200, B200, B300, MI300X, MI325X, MI355X.
  • moe_small: H200, B300, MI300X, MI325X, MI355X.
  • Counters mode: H200, B200, MI300X, MI325X, MI355X.
  • B300 counters needs Nsight Compute ≥ 2025.3 on the dsxe nodes.
  • MI355X moe_small reports a failure only because one shard is all-unsupported under --strict; its data is complete.
• tests/test_ci.py passes (32).

🤖 Generated with Claude Code

[claude]

Beyond the inline findings, I also checked the AMD torch GEMM backend's fp8 unsupported-case handling (runners/amd/backends/torch.py) against the actual testlist data: none of the e4m3 entries in gemm.json, gemm_perf.json, or the gemm_serving_*_min.json lists use the static per-tensor scale format the backend supports — they're all block-scaled (group [128,128] etc.), which the backend correctly rejects as unsupported. So this is the backend's documented per-tensor-only scope matching the current data, not a regression introduced by this change.

Extended reasoning...

This PR rewrites the operatorx microbenchmark tool end to end (CI workflow, core op/backend framework, gemm/moe op specs, and every vendor runner backend for NVIDIA/AMD/TPU/Trainium); it touches CI dispatch/scheduling but no auth or crypto surface. Four confirmed findings are already queued as inline comments (a Trainium NEFF-selection fallback that silently degrades to noisy host timing, testlist entries that bypass schema validation entirely, a telemetry polling-thread leak on kernel fault, and an eager registry-digest lookup in ci.py's plan() that can fail a whole plan on one transient pruned-image error). I additionally verified, by grepping the regenerated gemm testlists for the static per-tensor fp8 scale shape the AMD torch backend accepts, that no current test case actually hits that accept path with a genuine per-tensor scheme — the backend's narrow fp8 support matches the shipped data rather than silently dropping real coverage.

Findings marked 🟡 are optional suggestions and need no follow-up push.

[functionstackx]

Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding