Skip to content

Commit fde20d9

Browse files
[PyBindings] Add zero-copy DLPack interchange (#23090)
## PR stack #23088 adds Python buffer inputs, relanded in #23535. #23089 removes the portable bindings' C++ ATen dependency. This PR adds DLPack interchange on top of the updated #23089 head (`55f443a9f8`). ## Summary Accept dense CPU and CUDA DLPack producers alongside Python buffers and `torch.Tensor` inputs. Preserve pointers, dtype, shape, strides, storage offsets, and device indices while checking compatibility with ExecuTorch dimension order and serialized input devices. `ExecuTorchResult` exposes `__dlpack__` and `__dlpack_device__`. Outputs remain Python Torch tensors when Torch is available; `EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT=executorch` selects ExecuTorch results without importing Torch. When the setting is unset and Torch is unavailable, the bindings automatically select ExecuTorch results. An explicit `torch` setting still requires Torch to import successfully. CUDA outputs remain device allocations. Borrowed results retain their actual runtime or producer storage, and cloned CUDA results use device-to-device copies without a C++ ATen dependency. CUDA DLPack inputs and ExecuTorch CUDA results require CUDA-enabled bindings. Torch CUDA inputs and outputs synchronize through Python Torch, preserving their behavior in CPU builds of the bindings. The CUDA boundary currently synchronizes device work to establish producer, delegate, and consumer readiness. DLPack producers receive the supported legacy default stream. Retain installed Module inputs for subsequent executions, bound ownership during repeated result feedback, and serialize calls on each shared method and module. The Windows wheel uses its shipped CUDA helper without requiring an external CUDA runtime DLL at import. CUDA result cloning can use that helper's allocator without requiring CUDA delegate registration. CUDA CI selects a compatible C++ runtime so CUDA 12.6 can load its AOTI model libraries. ## Validation The CUDA build workflow runs three required-GPU integration tests in both output modes on CUDA 12.6, 13.0, and 13.4 runners. They exercise a real CUDA delegate, delayed producer streams, separate consumer streams, driver-verified device pointers, offsets, channels last, cloning, direct result roundtrip, and borrowed lifetimes. The identity test covers both Torch and DLPack inputs. Missing CUDA hardware or backend fails the CI invocation. Locally, nine focused CPU checks passed, including import policies, shared-object concurrency, DLPack inputs, layout validation, mixed protocols, runtime error codes, and ownership. Shared-object concurrency also passed in ExecuTorch output mode. Execution with the output variable unset passed in a Torch-free interpreter. Portable compilation and linking with LTO, CUDA helper compilation against CUDA 12.6 headers using GCC 13, workflow YAML, shell syntax, formatting, and diff checks passed. The local extension links no Torch libraries. GPU execution and Windows wheel verification await CI because this machine has no CUDA device or Windows runtime. Authored with OpenAI Codex.
1 parent a104810 commit fde20d9

17 files changed

Lines changed: 1903 additions & 147 deletions

File tree

‎.ci/scripts/test-cuda-build.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,14 @@ except Exception as e:
8787
exit(1)
8888
"
8989

90+
# Require the CUDA backend and hardware instead of silently skipping these
91+
# tests. Separate processes also exercise the output policy fixed at import.
92+
for tensor_output in torch executorch; do
93+
EXECUTORCH_PYBIND_TEST_REQUIRE_CUDA=1 \
94+
EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT="$tensor_output" \
95+
python extension/pybindings/test/test_pybindings_cuda_dlpack.py -v
96+
done
97+
9098
echo "SUCCESS: ExecuTorch CUDA ${cuda_version} build and verification completed successfully"
9199
}
92100

‎.ci/scripts/tests/test_cuda_workflow.py‎

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -51,17 +51,16 @@ def test_cuda_builds_take_the_node_driver_on_an_unpinned_v3(self):
5151
for pin in ("test-infra-ref", "driver-version", "driver-download-url"):
5252
self.assertNotIn(pin, job["with"])
5353

54-
def test_cuda134_runtime_update_precedes_build_and_propagates_failure(self):
54+
def test_cuda_runtime_update_precedes_build_and_propagates_failure(self):
5555
script = WORKFLOW["jobs"]["test-cuda-builds"]["with"]["script"]
5656
stubs = """
5757
conda() { printf 'CONDA %s\n' "$*"; return "$CONDA_STATUS"; }
58-
source() { printf 'BUILD %s\n' "$*"; }
58+
source() { printf 'BUILD %s\n' "$*"; printf 'LIBRARY PATH %s\n' "$LD_LIBRARY_PATH"; }
5959
"""
6060
for version, conda_status in (
61-
("12.6", 0),
62-
("13.0", 0),
63-
("13.4", 0),
64-
("13.4", 1),
61+
(version, status)
62+
for version in ("12.6", "13.0", "13.4")
63+
for status in (0, 1)
6564
):
6665
with self.subTest(version=version, conda_status=conda_status):
6766
result = subprocess.run(
@@ -70,19 +69,26 @@ def test_cuda134_runtime_update_precedes_build_and_propagates_failure(self):
7069
"-c",
7170
stubs + script.replace("${{ matrix.cuda-version }}", version),
7271
],
73-
env={**os.environ, "CONDA_STATUS": str(conda_status)},
72+
env={
73+
**os.environ,
74+
"CONDA_STATUS": str(conda_status),
75+
"LD_LIBRARY_PATH": "/cuda/lib64",
76+
},
7477
capture_output=True,
7578
text=True,
7679
)
7780
self.assertEqual(result.returncode, conda_status, result.stderr)
78-
expected = []
79-
if version == "13.4":
80-
expected.append(
81-
"CONDA install -y -n base -c conda-forge "
82-
"libstdcxx-ng=16.2.0 libgcc-ng=16.2.0"
83-
)
81+
expected = [
82+
"CONDA install -y -n base -c conda-forge "
83+
"libstdcxx-ng=16.2.0 libgcc-ng=16.2.0"
84+
]
8485
if conda_status == 0:
85-
expected.append(f"BUILD .ci/scripts/test-cuda-build.sh {version}")
86+
expected.extend(
87+
(
88+
f"BUILD .ci/scripts/test-cuda-build.sh {version}",
89+
"LIBRARY PATH /opt/conda/lib:/cuda/lib64",
90+
)
91+
)
8692
self.assertEqual(result.stdout.splitlines(), expected)
8793

8894
def test_cuda_probe_checks_the_result_and_torch_train(self):

‎.ci/scripts/wheel/test_cuda_windows.py‎

Lines changed: 21 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,11 @@
1010
The Windows CUDA wheel ships the CUDA delegate for C++ applications only. A CUDA program
1111
cannot be lowered on Windows, since lowering compiles the model with a toolchain that only
1212
the Linux side has, so a Windows user exports on Linux (or WSL) and runs the program with
13-
this wheel. The Python extension therefore carries no CUDA dependency at all, and the
13+
this wheel. The Python extension uses the shipped CUDA helper for tensor interchange, and the
1414
checks here hold both halves of that:
1515
1616
the CUDA DLLs ship, with import libraries a C++ consumer links
17-
the Python extension depends on no CUDA DLL, so importing the package needs no CUDA
17+
the Python extension needs no CUDA runtime DLL, so importing the package needs no CUDA
1818
the delegate registers in a C++ application linking executorch::backend_cuda
1919
the device code covers the GPUs the row claims
2020
a program exported on Linux for a Windows target runs through the C++ SDK and matches eager
@@ -209,30 +209,28 @@ def test_cuda_libraries_are_shipped() -> None:
209209
print(f"✓ the CUDA DLLs and their import libraries ship ({len(expected)} files)")
210210

211211

212-
def test_python_extension_carries_no_cuda() -> None:
213-
"""The Python extension must depend on no CUDA DLL.
214-
215-
CUDA programs are lowered on Linux, so the Windows extension has nothing to do with the
216-
delegate, and depending on it would make importing the package fail on a machine
217-
without the CUDA runtime.
218-
"""
212+
def test_python_extension_requires_no_cuda_runtime() -> None:
213+
"""The bindings and their shipped helper must import without a CUDA toolkit."""
219214
import test_shared_libraries
220215

221216
package = _package_dir()
222217
extensions = sorted((package / "extension" / "pybindings").glob("_C.*.pyd"))
223218
assert len(extensions) == 1, f"expected one _C extension, found {extensions}"
224-
dependents = {
225-
name.lower() for name in test_shared_libraries._pe_dependents(extensions[0])
226-
}
227-
cuda = sorted(
228-
name
229-
for name in dependents
230-
if "cuda" in name or name.startswith(("cudart", "cublas", "nvrtc"))
231-
)
232-
assert not cuda, (
233-
f"{extensions[0].name} depends on {cuda}, so the Python side of the Windows wheel "
234-
"carries CUDA it cannot use and fails to import without the CUDA runtime"
235-
)
219+
helper = package / "lib" / "executorch_extension_cuda.dll"
220+
for library in (extensions[0], helper):
221+
dependents = {
222+
name.lower() for name in test_shared_libraries._pe_dependents(library)
223+
}
224+
cuda = sorted(
225+
name
226+
for name in dependents
227+
if ("cuda" in name or name.startswith(("cudart", "cublas", "nvrtc")))
228+
and name != helper.name.lower()
229+
)
230+
assert not cuda, (
231+
f"{library.name} imports {cuda}, so importing the bindings would "
232+
"require CUDA libraries outside the wheel"
233+
)
236234
from executorch.extension.pybindings.portable_lib import (
237235
_get_registered_backend_names,
238236
)
@@ -243,7 +241,7 @@ def test_python_extension_carries_no_cuda() -> None:
243241
"Windows wheel does not ship it for"
244242
)
245243
print(
246-
f"✓ {extensions[0].name} depends on no CUDA DLL and registers no CUDA backend"
244+
f"✓ {extensions[0].name} requires no CUDA runtime DLL and registers no CUDA backend"
247245
)
248246

249247

@@ -628,7 +626,7 @@ def export(destination: str) -> None:
628626
test_clean_install.run_tests(Path(work_dir))
629627

630628
test_cuda_libraries_are_shipped()
631-
test_python_extension_carries_no_cuda()
629+
test_python_extension_requires_no_cuda_runtime()
632630
test_cuda_runtime_is_linked_statically()
633631
test_device_code_covers_the_row()
634632
with tempfile.TemporaryDirectory() as work_dir:

‎.github/workflows/cuda.yml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@ jobs:
120120
contains(needs.changed-files.outputs.changed-files, 'backends/cuda') ||
121121
contains(needs.changed-files.outputs.changed-files, 'backends/aoti') ||
122122
contains(needs.changed-files.outputs.changed-files, 'extension/cuda') ||
123+
contains(needs.changed-files.outputs.changed-files, 'extension/pybindings') ||
123124
contains(needs.changed-files.outputs.changed-files, '.github/workflows/cuda.yml') ||
124125
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test-cuda-build.sh') ||
125126
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/export_model_artifact.sh') ||
@@ -145,10 +146,10 @@ jobs:
145146
script: |
146147
set -eux
147148
148-
if [ "${{ matrix.cuda-version }}" = "13.4" ]; then
149-
# The image's older Conda runtime aborts during CUDA thread-local cleanup.
150-
conda install -y -n base -c conda-forge 'libstdcxx-ng=16.2.0' 'libgcc-ng=16.2.0'
151-
fi
149+
# AOTI model libraries need a newer C++ runtime than the 12.6 image's
150+
# system library; the 13.4 image also needs the thread-local cleanup fix.
151+
conda install -y -n base -c conda-forge 'libstdcxx-ng=16.2.0' 'libgcc-ng=16.2.0'
152+
export LD_LIBRARY_PATH="/opt/conda/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
152153
source .ci/scripts/test-cuda-build.sh "${{ matrix.cuda-version }}"
153154
154155
# This job will fail if any of the CUDA versions fail
@@ -164,6 +165,7 @@ jobs:
164165
contains(needs.changed-files.outputs.changed-files, 'backends/cuda') ||
165166
contains(needs.changed-files.outputs.changed-files, 'backends/aoti') ||
166167
contains(needs.changed-files.outputs.changed-files, 'extension/cuda') ||
168+
contains(needs.changed-files.outputs.changed-files, 'extension/pybindings') ||
167169
contains(needs.changed-files.outputs.changed-files, '.github/workflows/cuda.yml') ||
168170
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test-cuda-build.sh') ||
169171
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/export_model_artifact.sh') ||

‎.github/workflows/pull.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,17 @@ jobs:
176176
python -m examples.portable.scripts.export \
177177
--model_name=add \
178178
--output_dir=cmake-out-no-aten
179+
python - <<'PY'
180+
from pathlib import Path
181+
182+
from executorch.extension.pybindings.test.make_test import (
183+
create_program,
184+
ModuleAddBFloat16,
185+
)
186+
187+
program, _ = create_program(ModuleAddBFloat16())
188+
Path("cmake-out-no-aten/add_bfloat16.pte").write_bytes(bytes(program.buffer))
189+
PY
179190
EXECUTORCH_PYBIND_TEST_PTE=cmake-out-no-aten/add.pte \
180191
PYTHONPATH=cmake-out-no-aten:.. \
181192
python extension/pybindings/test/test_pybindings_no_aten.py \
@@ -187,11 +198,13 @@ jobs:
187198
PybindingsNoAtenTest.test_executorch_output_is_sticky_after_torch_import
188199
python -m pip uninstall -y torch torchvision torchaudio
189200
EXECUTORCH_PYBIND_TEST_PTE=cmake-out-no-aten/add.pte \
201+
EXECUTORCH_PYBIND_TEST_BFLOAT16_PTE=cmake-out-no-aten/add_bfloat16.pte \
190202
PYTHONPATH=cmake-out-no-aten:.. \
191203
python extension/pybindings/test/test_pybindings_no_aten.py \
192204
PybindingsNoAtenTest.test_output_policy_without_torch
193205
EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT=executorch \
194206
EXECUTORCH_PYBIND_TEST_PTE=cmake-out-no-aten/add.pte \
207+
EXECUTORCH_PYBIND_TEST_BFLOAT16_PTE=cmake-out-no-aten/add_bfloat16.pte \
195208
PYTHONPATH=cmake-out-no-aten:.. \
196209
python extension/pybindings/test/test_pybindings_no_aten.py
197210
if ldd cmake-out-no-aten/_C*.so | grep -Eiq 'torch|aten|c10'; then

‎CMakeLists.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1457,6 +1457,10 @@ if(EXECUTORCH_BUILD_PYBIND)
14571457
target_compile_options(portable_lib PRIVATE -fvisibility-inlines-hidden)
14581458
endif()
14591459
target_link_libraries(portable_lib PRIVATE ${_dep_libs})
1460+
if(TARGET extension_cuda)
1461+
target_link_libraries(portable_lib PRIVATE extension_cuda)
1462+
target_compile_definitions(portable_lib PRIVATE EXECUTORCH_PYBINDINGS_CUDA)
1463+
endif()
14601464
executorch_target_link_shared_runtime(portable_lib)
14611465
# These libraries register their operators or their backend from a static
14621466
# initializer, so nothing here references a symbol from them and some linkers

‎extension/pybindings/BUCK‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,11 @@ fbcode_target(_kind = runtime.export_file,
1616
visibility = ["//executorch/extension/pybindings/..."],
1717
)
1818

19+
fbcode_target(_kind = runtime.export_file,
20+
name = "pybindings_cuda.h",
21+
visibility = ["//executorch/extension/pybindings/..."],
22+
)
23+
1924
fbcode_target(_kind = runtime.export_file,
2025
name = "pybindings_dlpack.h",
2126
visibility = ["//executorch/extension/pybindings/..."],

‎extension/pybindings/README.md‎

Lines changed: 60 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -13,24 +13,32 @@ pip install . --no-build-isolation
1313
Portable bindings do not link against ATen or PyTorch. CPU inputs may be NumPy
1414
arrays or other dense objects implementing Python's buffer protocol. CPU and
1515
CUDA `torch.Tensor` inputs are inspected through their public Python attributes
16-
instead of being cast to `at::Tensor`; CUDA pointers retain their device index.
16+
instead of being cast to `at::Tensor`. DLPack producers, including CuPy, can
17+
provide CPU or CUDA allocations directly; their device indices are preserved.
18+
Tensor protocols can be mixed within the same call.
1719

18-
Portable Torch inputs must have dense strides compatible with the exported
19-
method input's dimension order. This includes channels last and channels last
20-
3D layouts when the model was exported with those layouts. Strides on singleton
21-
dimensions are normalized to the exported layout without copying tensor data.
20+
Tensor inputs must have dense storage laid out for the exported method's
21+
dimension order. Stride compatibility checks are deferred; the bindings interpret
22+
Torch and DLPack tensor storage using the exported layout without copying data.
23+
CUDA inputs require a method
24+
exported with CUDA inputs and `MemoryPlanningPass(alloc_graph_input=False)` so
25+
execution can share the caller's device allocation.
2226

2327
When PyTorch is available, importing `portable_lib` imports it and portable
24-
bindings create output tensors through its public Python API. When it cannot be
25-
imported, the default output type is `ExecuTorchResult`. The extension itself does not link against ATen or PyTorch.
26-
CUDA outputs use a private DLPack capsule consumed by `torch.from_dlpack`, and
27-
`clone_outputs=True` copies them through Torch on their original CUDA device.
28-
The `executorch` output mode supports CPU buffers; CUDA outputs require the
29-
default Torch output mode.
28+
bindings return `torch.Tensor` outputs. When it cannot be imported, the default
29+
output type is `ExecuTorchResult`. Output conversion uses Python APIs and
30+
DLPack; the extension does not link against PyTorch.
31+
`clone_outputs=True` returns independent storage on the output's original
32+
device. With `clone_outputs=False`, results share runtime or input storage and
33+
later execution may overwrite that storage.
34+
35+
Calls on a shared method or module are serialized. A sequence of separate
36+
`set_inputs`, `execute`, and `get_outputs` calls requires caller synchronization
37+
when each thread needs its own inputs and outputs.
3038

3139
To select `ExecuTorchResult` outputs even when PyTorch is installed, set the
32-
environment variable before importing the bindings. The setting is fixed when the extension loads, so importing PyTorch
33-
later does not change the output type.
40+
environment variable before importing the bindings. The setting is fixed when
41+
the extension loads, so importing PyTorch later does not change the output type.
3442

3543
```python
3644
import os
@@ -39,15 +47,48 @@ os.environ["EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT"] = "executorch"
3947
from executorch.extension.pybindings import portable_lib
4048
```
4149

42-
`ExecuTorchResult` exposes shape, strides, dtype, and the Python buffer protocol
43-
so NumPy can read it without another copy. Explicitly setting the environment
44-
variable to `torch` requires PyTorch to import successfully. ATen-mode bindings continue to return native PyTorch tensors directly.
50+
`ExecuTorchResult` preserves shape, strides, dtype, and device. CPU results expose
51+
the Python buffer protocol so NumPy can read them without another copy. Without
52+
the environment setting, the bindings fall back to `ExecuTorchResult` if
53+
PyTorch cannot be imported. Explicitly selecting `torch` requires it to import
54+
successfully. ATen-mode bindings continue to return native PyTorch tensors
55+
directly.
4556

4657
```python
4758
output = module.forward([numpy_input])[0]
4859
result = np.asarray(output)
4960
```
5061

62+
CPU and CUDA results implement `__dlpack__` and `__dlpack_device__`. CUDA results
63+
remain on CUDA and can be passed to another DLPack consumer or back into
64+
ExecuTorch without moving the data to the CPU.
65+
66+
CUDA DLPack inputs and `ExecuTorchResult` outputs require CUDA-enabled bindings.
67+
Torch CUDA inputs and outputs synchronize through Python Torch, including in
68+
CPU builds of the bindings. The Python CUDA execution boundary waits for device
69+
work because the device allocator does not expose readiness events.
70+
Execution is synchronous at this boundary, including producer and consumer
71+
stream readiness. The synchronization has no C++ ATen dependency.
72+
73+
For bfloat16 inputs, raw byte buffers use the shape declared by the exported
74+
method. Bfloat16 results expose flat bytes through the buffer protocol because
75+
NumPy and PEP 3118 do not define that numeric dtype. DLPack consumers receive the
76+
original bfloat16 dtype and tensor shape.
77+
78+
```python
79+
import cupy as cp
80+
81+
cuda_input = cp.ones((1, 3, 224, 224), dtype=cp.float32)
82+
output = cuda_module.forward([cuda_input], clone_outputs=False)[0]
83+
cuda_output = cp.from_dlpack(output)
84+
```
85+
86+
The CUDA build workflow runs `test_pybindings_cuda_dlpack.py` in both output
87+
modes on a GPU. It checks device allocations, DLPack stream ordering, channels
88+
last views, cloning, and borrowed output lifetimes using a CUDA delegate.
89+
Set `EXECUTORCH_PYBIND_TEST_REQUIRE_CUDA=1` when running this test to treat missing
90+
GPU hardware or a missing CUDA backend as a failure.
91+
5192
# Link Backends
5293

5394
Not all backends are built into the pip wheel by default. You can link these missing/experimental backends by turning on the corresponding cmake flag. For example, to include the Vulkan backend:
@@ -68,9 +109,8 @@ CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON" ./install_executorch.sh
68109
## Classes
69110
### ExecuTorchModule
70111
- `plan_execute()`: Plan and execute.
71-
- `run_method()`: Run a method with either PyTorch tensors or objects that
72-
implement Python's buffer protocol, such as NumPy arrays. A call must use a
73-
single tensor protocol; PyTorch tensors and buffers cannot be mixed.
112+
- `run_method()`: Run a method with PyTorch tensors, Python buffer objects such
113+
as NumPy arrays, or DLPack producers, including mixed tensor protocols.
74114
- `forward()`: Forward. A single buffer may be passed directly; multiple
75115
inputs are passed as a flat sequence.
76116
- `has_etdump()`: Check if etdump is available.
@@ -80,6 +120,4 @@ CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON" ./install_executorch.sh
80120
This class is currently empty and serves as a placeholder for future methods and attributes.
81121
- `verify_result_with_bundled_expected_output(method_name: str, testset_idx: int, rtol: float = 1e-5, atol: float = 1e-8)`: Verify result with bundled expected output.
82122
## Note
83-
Calls on a shared method or module are serialized. A sequence of separate
84-
`set_inputs`, `execute`, and `get_outputs` calls requires caller synchronization
85-
when each thread needs its own inputs and outputs.
123+
Runtime logs are forwarded to Python stderr.

0 commit comments

Comments
 (0)