Skip to content

backend_dispatch.rst

Latest commit

 

History

History
129 lines (104 loc) · 5.88 KB

backend_dispatch.rst

File metadata and controls

129 lines (104 loc) · 5.88 KB
Raw

Backend Dispatch

The glass::nvidia::block::gemm / gemv / row_strided_* / gemm_batched_1d primary templates auto-dispatch at compile time: for shapes where pure-SIMT wins they fall through to ::glass::*; for shapes where the vendor library wins they route to cuBLASDx via the DEFINE_NVIDIA_* macros.

This means glass::nvidia::block::gemm<float, 6, 6, 6>(...) "just works" without any DEFINE macro — small shapes route to SIMT automatically. Larger shapes such as glass::nvidia::block::gemm<float, 32, 32, 32>(...) still require a DEFINE_NVIDIA_GEMM(32, 32, 32) in scope (placed inside namespace glass { namespace nvidia { namespace block {), but produce a clean compile-time message when it is missing.

Note

This cuBLASDx-vs-SIMT decision is one of three distinct dispatch layers. glass::recommend<> (:doc:`tuning`) is a host-side ladder that advises launch-level packing — which native or NVIDIA thread / warp / block tier to launch, i.e. the shape of <<<grid, block>>>. Newest, and different from both: glass::dispatch_body() (glass-dispatch.cuh) picks the in-block body behind the bare glass::op face, under a fixed block-scope calling contract — the launch does not change. The measured in-block body sweep (bench/tune.py --legs body, Phase 2, 2026-07-30) moved the winning cells to a warp- or thread-body executed inside the block — an attested, receipt-gated retune, regenerated per arch like the ladder. A moved cell matches the block body to tolerance, not bit-exactly; ops with no moved cell remain exactly glass::block::. Pin glass::block:: where determinism is load-bearing; see :doc:`namespaces` and :doc:`tuning`.

The dispatch flow

           caller writes:  glass::nvidia::block::gemm<float, M, N, K>(...)
                                  │
                                  ▼
                should_use_cublasdx<float, M, N, K, SMS>()
                                  │
           ┌──────────────────────┴──────────────────────┐
           ▼                                             ▼
         false                                          true
           │                                             │
────────── ▼ ──────────              ────────── ▼ ──────────
SIMT fallback:                       Need a DEFINE_NVIDIA_GEMM*
::glass::gemm<T,M,N,K>(...)          to specialize for cuBLASDx;
(no DEFINE needed; no smem)          else: static_assert error.

The decision is made at compile time by should_use_cublasdx*<T,M,N,K,SM>() (see src/nvidia/query.cuh / query_simt.cuh), which consults — in order:

  1. A per-build local override table (when GLASS_TUNING_TABLE_LOCAL is defined).
  2. The shipped global table (src/nvidia/tuning_table.cuh).
  3. A fallback static heuristic.

The shipped table currently covers sm_120 (Blackwell-class) for square shapes from 3×3×3 up to 64×64×64; regenerate for another arch with bench/autotune.py (a new arch gets its own entries without touching the existing ones).

The size heuristic

When no tuning-table entry matches, each API falls back to a per-API heuristic that reflects its arithmetic intensity:

Template Default heuristic
cublasdx_wins<M, N, K, SM> (gemm) max(M,N,K) >= 16 AND min(M,N,K) >= 4
cublasdx_wins_gemv<M, N, SM> max(M,N) >= 32
cublasdx_wins_batched<M, N, K, BATCH, SM> BATCH >= 8 AND max(M,N,K) >= 8
cublasdx_wins_gemm_strided<...> delegates to cublasdx_wins<>
cublasdx_wins_gemv_strided<...> delegates to cublasdx_wins_gemv<>

Restricted to float: the heuristic returns SIMT for non-float types.

Inspecting the decision

The print_dispatch* host helpers (__host__ __device__, so you can also drop one into a kernel for runtime diagnostics) report the chosen path:

glass::nvidia::block::print_dispatch<float, 4, 4, 4>();
// → glass::nvidia::block::gemm<T,4,4,4,SM=860>: SIMT fallback

glass::nvidia::block::print_dispatch<float, 32, 32, 32>();
// → glass::nvidia::block::gemm<T,32,32,32,SM=860>: cuBLASDx (needs DEFINE_NVIDIA_GEMM*)

glass::nvidia::block::print_dispatch_gemv<float, 64, 64>();
// → glass::nvidia::block::gemv<T,64,64,SM=860>: cuBLASDx (needs DEFINE_NVIDIA_GEMV*)

Overriding the dispatch

Goal How
Force cuBLASDx for a shape the heuristic puts in SIMT Add DEFINE_NVIDIA_GEMM(M,N,K) in your .cu file — the explicit specialization always overrides the primary template.
Force SIMT for a shape the heuristic puts in cuBLASDx Call ::glass::block::gemm<T,M,N,K>(...) directly (skip the NVIDIA path).
Per-host tuning without editing source Run python bench/autotune.py to generate bench/tuning/<hostname>.cuh, then compile with -DGLASS_TUNING_TABLE_LOCAL='"bench/tuning/<hostname>.cuh"'.
Different SM in-tree (for a PR) python bench/autotune.py --in-tree rewrites the marker-delimited specializations inside src/nvidia/tuning_table.cuh.

The shipped values are sensible defaults, but small-GEMM performance is highly SM-dependent. See :doc:`tuning` for the full per-host autotuning workflow that measures both paths side by side on your hardware so you can override these defaults.