Skip to content

Commit bfb3273

Browse files
authored
feat: isolate cutlass._mlir imports behind compat gateway (#118) (#121)
* feat: isolate cutlass._mlir imports behind _mlir_compat gateway (#118) All 10 kernel modules previously imported CuTeDSL's private generated bindings (cutlass._mlir.{ir,arith,llvm,vector,nvvm,cute}) directly, so a CutDSL patch release could break kernel emission silently (the 4.5.2->4.5.3 tcgen05_ld/st incident). Add a single-point gateway that lazily loads the private dialects, enforces the pyproject.toml version contract (including the !=4.5.0 exclusion), and probes canary entry points, failing fast with an actionable error. Migrate all consumers to bind their dialect aliases from the gateway. Extend test_cutedsl_compat.py with fault-injection and version-matrix tests that run headless. * fix: cross-version vector element extraction in store_256b CutDSL renamed vector.extractelement to vector.extract in the 4.6 line, which broke store_256b (used by KDA SM100 backward) against nvidia-cutlass-dsl 4.6.2 at JIT time. Route element extraction through the gateway's version-dispatching vector_extract_element helper. * chore: add Modal GPU validation harness (H100, CUDA 12.9) * fix: address review comments on mlir compat gateway (#121) - vector_extract_element: prefer extractelement (builds the i32 index constant internally) and use the static-position extract(vec, [], [pos]) form on 4.6+, matching the real generated bindings - normalize parsed versions to (major, minor, patch) so '4.5'/'4.7' hit the exclusion and upper bound instead of slipping past - modal_validate: use the module-level TESTS list, pass args without shell=True, and fail the Modal function on nonzero pytest exit - keep ruff lint/format clean (CI runs ruff --all-files) * feat: parametrize modal harness by GPU and CuTeDSL version Adds test_ptx_umma_ws.py (SM100, self-deselects on non-Blackwell via conftest) to the validation suite so the store_256b vector-extract path the reviewer flagged on GB200 is covered, and lets the harness run any GPU/CuTeDSL combination: modal run scripts/modal_validate.py --gpu B200 --cutlass 'nvidia-cutlass-dsl==4.5.2' * feat: parametrize modal harness via env (client 1.4.2 has no with_options) * fix: harness must run sm100 tests on Blackwell and report the real GPU The '-m not sm100_only' marker expression deselected the SM100 tests even on B200 (conftest already skips them on non-Blackwell); the GPU was also hardcoded in the summary because env vars do not reach the container, so pass it as an argument. * style: apply ruff 0.15.0 import sort and format fixes
1 parent 5161546 commit bfb3273

13 files changed

Lines changed: 623 additions & 29 deletions

File tree

cula/lightning/la_verify_kvbuffer.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,6 @@
4949
import cutlass
5050
import cutlass.cute as cute
5151
import torch
52-
from cutlass._mlir.dialects import arith as _arith
53-
from cutlass._mlir.dialects import llvm as _llvm
5452
from cutlass.cute.runtime import (
5553
make_fake_compact_tensor,
5654
make_fake_stream,
@@ -63,6 +61,8 @@
6361
NUM_THREADS_MTP,
6462
hq_dot_pair,
6563
)
64+
from cula.ops._mlir_compat import arith as _arith
65+
from cula.ops._mlir_compat import llvm as _llvm
6666
from cula.utils import USE_FAST_MATH, get_device_sm_version
6767

6868
# Dispatch threshold between the two verify implementations.

cula/ops/_mlir_compat.py

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
# Copyright 2025-2026 Ant Group Co., Ltd.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Single-point gateway for CuTeDSL's private MLIR/NVVM bindings.
16+
17+
cuLA kernels are written against CuTeDSL's code-generation API. Most of that API
18+
is public (``cutlass.cutlass_dsl``, ``cutlass.cute``, ...); a small part is not:
19+
the generated MLIR dialect bindings under ``cutlass._mlir``. CuTeDSL ships them
20+
as implementation detail and provides no stability contract across patch
21+
releases -- the ``tcgen05_ld/st`` breakage between CutDSL 4.5.2 and 4.5.3 was one
22+
such incident.
23+
24+
This module is the ONLY place in cuLA that may import from ``cutlass._mlir``.
25+
Kernel modules bind their dialect aliases from here::
26+
27+
from cula.ops._mlir_compat import arith as _arith
28+
from cula.ops._mlir_compat import ir
29+
from cula.ops._mlir_compat import llvm as _llvm
30+
from cula.ops._mlir_compat import vector as _vector
31+
32+
Design goals:
33+
34+
- Lazy: nothing is imported until a kernel actually needs a binding, so plain
35+
imports of cuLA never touch ``cutlass._mlir``.
36+
- Explicity: an out-of-contract CuTeDSL version, or a missing/renamed binding,
37+
fails fast at first use with an actionable ``RuntimeError`` instead of
38+
surfacing mid-JIT as a confusing compile error (or silently emitting a
39+
different kernel).
40+
- Zero dependencies: version parsing is done with a small regex so this module
41+
stays usable in any environment cuLA can be installed into.
42+
43+
The version contract mirrors ``pyproject.toml`` (including its ``!=4.5.0``
44+
exclusion) and the canary probes make a broken binding fail fast with an
45+
actionable message instead of surfacing mid-JIT.
46+
"""
47+
48+
from __future__ import annotations
49+
50+
import importlib
51+
import re
52+
from typing import Any, Final
53+
54+
# Version contract for nvidia-cutlass-dsl. Kept in sync with
55+
# ``pyproject.toml``; when CuTeDSL is bumped, extend ``_SUPPORTED_MIN`` /
56+
# ``_SUPPORTED_MAX`` only after the new release has been validated against the
57+
# canaries below (and ideally against the SM90/SM100 kernel test suites).
58+
_SUPPORTED_MIN: Final[tuple[int, ...]] = (4, 4, 2)
59+
_SUPPORTED_MAX: Final[tuple[int, ...]] = (4, 7, 0)
60+
_EXCLUDED_VERSIONS: Final[frozenset[tuple[int, ...]]] = frozenset({(4, 5, 0)})
61+
62+
# dialect name -> (package, attribute) inside ``cutlass``.
63+
_PRIVATE_TABLE: Final[dict[str, tuple[str, str]]] = {
64+
"arith": ("cutlass._mlir", "dialects.arith"),
65+
"cute": ("cutlass._mlir", "dialects.cute"),
66+
"ir": ("cutlass._mlir", "ir"),
67+
"llvm": ("cutlass._mlir", "dialects.llvm"),
68+
"nvvm": ("cutlass._mlir", "dialects.nvvm"),
69+
"vector": ("cutlass._mlir", "dialects.vector"),
70+
}
71+
72+
# Canary entry points: attribute paths that must be present on each dialect
73+
# binding for cuLA's kernel code to be emitted correctly. These are the names
74+
# the migrated consumers actually call; extend the list when new usages land.
75+
_CANARIES: Final[dict[str, tuple[tuple[str, ...], ...]]] = {
76+
"arith": (("constant",),),
77+
"cute": (),
78+
"ir": (("Type", "parse"), ("VectorType", "get")),
79+
"llvm": (("inline_asm",), ("extractvalue",)),
80+
"nvvm": (),
81+
"vector": (("bitcast",), ("extract_strided_slice",)),
82+
}
83+
84+
# Entry points whose name changed across CutDSL versions: for each group (a
85+
# tuple of variant paths), at least one variant must exist. For example
86+
# ``vector.extractelement`` was replaced by ``vector.extract`` in the 4.6
87+
# line; cuLA helpers dispatch on whichever one exists.
88+
_ANY_OF_CANARIES: Final[dict[str, tuple[tuple[tuple[str, ...], ...], ...]]] = {
89+
"vector": ((("extract",), ("extractelement",)),),
90+
}
91+
92+
_VERSION_RE: Final[re.Pattern[str]] = re.compile(r"^(\d+)(?:\.(\d+))?(?:\.(\d+))?(?:[a-z0-9._+-]*)?$")
93+
94+
_INCIDENT_NOTE: Final[str] = (
95+
"cuLA depends on CuTeDSL's private MLIR bindings (`cutlass._mlir`), which are "
96+
"implementation detail and were broken by a CuTeDSL patch release once "
97+
"already (`tcgen05_ld/st`, 4.5.2 -> 4.5.3). To avoid silent kernel "
98+
"miscompiles, cuLA refuses bindings outside its validated contract."
99+
)
100+
101+
_CACHE: Final[dict[str, Any]] = {}
102+
103+
104+
def _parse_version(version: str) -> tuple[int, int, int]:
105+
"""Parse ``X.Y.Z[.devN...]`` into a comparable ``(major, minor, patch)`` tuple.
106+
107+
Missing components normalize to zero so that ``4.5`` and ``4.7`` compare
108+
against the contract exactly like ``4.5.0`` and ``4.7.0`` (a two-component
109+
``4.7`` must not slip under the ``< 4.7.0`` upper bound).
110+
"""
111+
match = _VERSION_RE.match(version.strip())
112+
if match is None:
113+
raise RuntimeError(
114+
f"cuLA cannot parse the installed CuTeDSL version {version!r}; "
115+
f"refusing to use its private MLIR bindings. {_INCIDENT_NOTE}"
116+
)
117+
major, minor, patch = (int(part) if part is not None else 0 for part in match.groups())
118+
return (major, minor, patch)
119+
120+
121+
def _installed_version() -> tuple[int, int, int]:
122+
try:
123+
import cutlass # noqa: PLC0415
124+
except ImportError:
125+
raise RuntimeError(
126+
"cuLA requires the nvidia-cutlass-dsl package; install it with `pip install 'nvidia-cutlass-dsl>=4.4.2,<4.7'`."
127+
) from None
128+
version = getattr(cutlass, "__version__", None)
129+
if not isinstance(version, str):
130+
raise RuntimeError(
131+
f"CuTeDSL is installed but exposes no `__version__` (got {version!r}); "
132+
f"refusing to use its private MLIR bindings. {_INCIDENT_NOTE}"
133+
)
134+
return _parse_version(version)
135+
136+
137+
def _check_contract(version: tuple[int, ...]) -> None:
138+
if version in _EXCLUDED_VERSIONS:
139+
raise RuntimeError(
140+
f"Installed CuTeDSL version {'.'.join(map(str, version))} is explicitly "
141+
f"excluded by cuLA (see pyproject.toml). {_INCIDENT_NOTE}"
142+
)
143+
if not (_SUPPORTED_MIN <= version < _SUPPORTED_MAX):
144+
raise RuntimeError(
145+
f"Installed CuTeDSL version {'.'.join(map(str, version))} is outside the "
146+
f"range validated by cuLA ({'.'.join(map(str, _SUPPORTED_MIN))} to "
147+
f"{'.'.join(map(str, _SUPPORTED_MAX))}, exclusive). "
148+
f"{_INCIDENT_NOTE} To proceed, pin the validated range in "
149+
f"pyproject.toml and re-validate the canaries in this module."
150+
)
151+
152+
153+
def _has_attribute_path(module: Any, path: tuple[str, ...]) -> bool:
154+
owner = module
155+
for part in path:
156+
owner = getattr(owner, part, None)
157+
if owner is None:
158+
return False
159+
return True
160+
161+
162+
def _load(dialect: str) -> Any:
163+
if dialect in _CACHE:
164+
return _CACHE[dialect]
165+
166+
package, attribute = _PRIVATE_TABLE[dialect]
167+
try:
168+
module = importlib.import_module(package)
169+
except ImportError as exc:
170+
raise RuntimeError(
171+
f"Unable to import CuTeDSL's private {dialect!r} bindings ({package}): {exc}. {_INCIDENT_NOTE}"
172+
) from exc
173+
if attribute:
174+
for part in attribute.split("."):
175+
module = getattr(module, part, None)
176+
if module is None:
177+
break
178+
if module is None:
179+
raise RuntimeError(f"CuTeDSL no longer exposes {dialect!r} bindings ({package}.{attribute}). {_INCIDENT_NOTE}")
180+
181+
for canary in _CANARIES[dialect]:
182+
owner: Any = module
183+
for part in canary:
184+
owner = getattr(owner, part, None)
185+
if owner is None:
186+
raise RuntimeError(
187+
f"CuTeDSL dialect {dialect!r} is missing the canary entry point "
188+
f"{'.'.join(canary)} used by cuLA kernels. {_INCIDENT_NOTE}"
189+
)
190+
for group in _ANY_OF_CANARIES.get(dialect, ()):
191+
if not any(_has_attribute_path(module, variant) for variant in group):
192+
raise RuntimeError(
193+
f"CuTeDSL dialect {dialect!r} exposes none of the entry points "
194+
f"{' / '.join('.'.join(variant) for variant in group)} expected by "
195+
f"cuLA kernels. {_INCIDENT_NOTE}"
196+
)
197+
198+
_CACHE[dialect] = module
199+
return module
200+
201+
202+
def __getattr__(name: str) -> Any:
203+
"""Lazy, contract-checked access to private dialect bindings."""
204+
if name not in _PRIVATE_TABLE:
205+
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
206+
version = _installed_version()
207+
_check_contract(version)
208+
return _load(name)
209+
210+
211+
def cutlass_dsl_version() -> str | None:
212+
"""Installed CuTeDSL version string, or None when not installed."""
213+
try:
214+
import cutlass # noqa: PLC0415
215+
except ImportError:
216+
return None
217+
version = getattr(cutlass, "__version__", None)
218+
return version if isinstance(version, str) else None
219+
220+
221+
def vector_extract_element(vec, position, *, loc=None, ip=None):
222+
"""Extract one element of ``vec`` at ``position``, across CutDSL versions.
223+
224+
CutDSL renamed ``vector.extractelement`` to ``vector.extract`` in the 4.6
225+
line. ``position`` is the Python element index; each branch builds the
226+
operand shape its binding actually expects:
227+
228+
- ``extractelement`` (4.5 line; also present in early 4.6): takes the
229+
index as a single i32 operand, so the constant is constructed here.
230+
- ``extract`` (4.6+): takes a sequence of index-typed dynamic operands
231+
plus a static-position array, so a constant position is ``extract(vec,
232+
[], [position], ...)``.
233+
234+
Preferring ``extractelement`` when both exist keeps the pre-4.6 code path
235+
byte-identical to what cuLA shipped before the gateway.
236+
"""
237+
vector_dialect = _load("vector")
238+
if _has_attribute_path(vector_dialect, ("extractelement",)):
239+
i32_ty = _load("ir").IntegerType.get_signless(32)
240+
index = _load("arith").constant(i32_ty, position, loc=loc, ip=ip)
241+
return vector_dialect.extractelement(vec, position=index, loc=loc, ip=ip)
242+
return vector_dialect.extract(vec, [], [position], loc=loc, ip=ip)

cula/ops/kda/decode/mtp_conv.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,11 @@
3333
import cutlass
3434
import cutlass.cute as cute
3535
import torch
36-
from cutlass._mlir.dialects import llvm as _llvm
3736
from cutlass.cute.runtime import from_dlpack
3837
from cutlass.cute.typing import Int32
3938
from cutlass.cutlass_dsl import T as _T
4039

40+
from cula.ops._mlir_compat import llvm as _llvm
4141
from cula.ops.kda.decode.cute import (
4242
TILE_K,
4343
_get_cached_stream,

cula/ops/kda/decode/mtp_kvbuffer.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1097,11 +1097,12 @@ def kda_decode_mtp_shuffle_kvbuffer(
10971097
# C/D [16,8] f32: c0=C[gid][2tig] c1=C[gid][2tig+1] c2=C[gid+8][2tig] c3=C[gid+8][2tig+1]
10981098
# ===========================================================================
10991099

1100-
from cutlass._mlir.dialects import arith as _arith # noqa: E402
1101-
from cutlass._mlir.dialects import llvm as _llvm # noqa: E402
11021100
from cutlass.cutlass_dsl import T as _T # noqa: E402
11031101
from cutlass.cutlass_dsl import dsl_user_op # noqa: E402
11041102

1103+
from cula.ops._mlir_compat import arith as _arith # noqa: E402
1104+
from cula.ops._mlir_compat import llvm as _llvm # noqa: E402
1105+
11051106

11061107
@dsl_user_op
11071108
def _mma_m16n8k8_tf32(a0, a1, a2, a3, b0, b1, c0, c1, c2, c3, *, loc=None, ip=None):

cula/ops/kda/sm100/delta_h.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,14 +27,14 @@
2727
import torch
2828
import torch.nn.functional as F
2929
import triton
30-
from cutlass._mlir.dialects import llvm as _llvm
3130
from cutlass.cute.nvgpu import cpasync, tcgen05
3231
from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream
3332
from cutlass.cute.typing import Float32, Int32, Int64
3433
from cutlass.cutlass_dsl import T as _T
3534
from fla.ops.utils import prepare_chunk_indices, prepare_lens
3635
from fla.utils import tensor_cache
3736

37+
from cula.ops._mlir_compat import llvm as _llvm
3838
from cula.ops.kda.sm100.policy import sm100_intracard_cp_decision
3939
from cula.utils import USE_FAST_MATH, assert_blackwell, get_device_sm_count
4040

cula/ops/kda/sm90/_common.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,10 @@
66
import cutlass
77
import torch
88
from cutlass import Int32
9-
from cutlass._mlir.dialects import llvm as _llvm
109
from cutlass.cutlass_dsl import T as _T
1110

11+
from cula.ops._mlir_compat import llvm as _llvm
12+
1213

1314
def _stream_key(device: torch.device) -> tuple[str, int]:
1415
return str(device), int(torch.cuda.current_stream(device).cuda_stream)

cula/ops/lightning/prefill_sm100.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,12 +58,12 @@
5858
import cutlass.utils as utils
5959
import cutlass.utils.blackwell_helpers as sm100_utils
6060
import torch
61-
from cutlass._mlir.dialects import llvm as _llvm
6261
from cutlass.cute.nvgpu import cpasync, tcgen05
6362
from cutlass.cute.runtime import make_fake_compact_tensor, make_fake_stream
6463
from cutlass.cute.typing import Float32, Int32, Int64
6564
from cutlass.cutlass_dsl import T as _T
6665

66+
from cula.ops._mlir_compat import llvm as _llvm
6767
from cula.utils import USE_FAST_MATH, assert_blackwell
6868

6969

cula/ops/lightning/sm90/prefill_kernel.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,16 +25,17 @@
2525

2626
import cuda.bindings.driver as cuda
2727
import cutlass
28-
import cutlass._mlir.dialects.cute as _cute_ir
2928
import cutlass.cute as cute
3029
import cutlass.cute.nvgpu.warpgroup as warpgroup
3130
import cutlass.pipeline as pipeline
3231
import cutlass.utils as utils
3332
import cutlass.utils.hopper_helpers as sm90_utils
34-
from cutlass._mlir.dialects import llvm
3533
from cutlass.cute.nvgpu import cpasync, warp
3634
from cutlass.utils.tensormap_manager import TensorMapManager, TensorMapUpdateMode
3735

36+
from cula.ops._mlir_compat import cute as _cute_ir
37+
from cula.ops._mlir_compat import llvm
38+
3839
from .schedule import (
3940
DYNAMIC_SMEM_ESTIMATE_BYTES,
4041
EPILOGUE_THREADS,

cula/ops/lightning/sm90/schedule.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,10 @@
2424

2525
import cutlass
2626
import cutlass.cute as cute
27-
from cutlass._mlir.dialects import llvm
2827
from cutlass.cutlass_dsl import T
2928

29+
from cula.ops._mlir_compat import llvm
30+
3031
TARGET_ARCH = "sm_90a"
3132

3233
THREADS_PER_WARP_GROUP = 128

cula/ops/ptx.py

Lines changed: 5 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,13 @@
1616

1717
import cutlass
1818
import cutlass.cute as cute
19-
from cutlass._mlir import ir
20-
from cutlass._mlir.dialects import arith as _arith
21-
from cutlass._mlir.dialects import llvm as _llvm
22-
from cutlass._mlir.dialects import vector as _vector
2319
from cutlass.cutlass_dsl import T as _T
2420
from cutlass.cutlass_dsl import dsl_user_op
2521

22+
from cula.ops._mlir_compat import ir, vector_extract_element
23+
from cula.ops._mlir_compat import llvm as _llvm
24+
from cula.ops._mlir_compat import vector as _vector
25+
2626

2727
def _to_ir(v, loc=None, ip=None):
2828
if hasattr(v, "ir_value"):
@@ -125,17 +125,8 @@ def store_256b(gmem_ptr, vec):
125125

126126
@dsl_user_op
127127
def _do(addr, v, *, loc=None, ip=None):
128-
i32_ty = ir.IntegerType.get_signless(32)
129128
ir_v = _to_ir(v, loc, ip)
130-
elems = [
131-
_vector.extractelement(
132-
ir_v,
133-
position=_arith.constant(i32_ty, i, loc=loc, ip=ip),
134-
loc=loc,
135-
ip=ip,
136-
)
137-
for i in range(8)
138-
]
129+
elems = [vector_extract_element(ir_v, i, loc=loc, ip=ip) for i in range(8)]
139130
operands = [_to_ir(addr, loc, ip)] + elems
140131
_llvm.inline_asm(
141132
ir.Type.parse("!llvm.void"),

0 commit comments

Comments
 (0)