Skip to content

Commit ab907e0

Browse files
authored
Move the CUDA allocator into extension/cuda so other delegates can share it (pytorch#23447)
Other CUDA delegates need ExecuTorch's allocator without compiling a second copy of its singleton. This moves the allocator into the shared CUDA extension and ships its public headers in the wheel. The class lives in `executorch::extension::cuda`. The old namespace, header and Buck target remain as compatibility aliases. CMake exports the required C10 definition and supplies CUDA toolkit headers when a toolkit is installed. The allocator header requires a CUDA development toolkit; `CUDAToolkit_ROOT` can select it. The stream and device guard headers remain usable without one. Allocation and registration behavior are unchanged by the move: the CUDA backend still registers the allocator, and linking only the extension does not. Delegates share the async pool and its retention policy. A cache trim can release unused blocks cached by another delegate, but does not invalidate live allocations. The library initializes its platform state before logging allocator errors. Buck no longer forces the whole library into every binary: it has no static registrar, and its one static initializer comes in with the allocator code that uses it. The allocator accepts any power of two alignment. Requests above 256 bytes use checked padding and pointer rounding. Deallocation frees the original pointer, and the move preserves this behavior. Validation: built and ran five CUDA C++ suites on Linux x86_64 with H100 and CUDA 12.8. All 76 cases passed without skips, including 21 allocator cases covering alignment, overflow, concurrency and the legacy alias. Allocator memory checking reported no errors or leaks, with reporting disabled for intentional invalid device API errors. Pinned formatting checks passed. Added checks for required wheel headers and a CMake allocator consumer. Windows CUDA with MSVC and ROCm were not built or run. Those jobs skip on fork pull requests. Buck linkage was not built; an executable target-contract check is not a Buck build.
1 parent 91b2e90 commit ab907e0

16 files changed

Lines changed: 335 additions & 192 deletions

File tree

‎.ci/scripts/wheel/test_cpp_sdk.py‎

Lines changed: 94 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1243,14 +1243,45 @@ def test_every_shipped_header_compiles(work_dir: Path) -> None:
12431243
# tests a configuration no consumer of this package is ever in.
12441244
"-DC10_USING_CUSTOM_GENERATED_MACROS",
12451245
]
1246-
# Deliberately no CUDA toolkit include directory. A CUDA wheel's own headers have to compile
1247-
# against nothing but the wheel, the same as every other header here. Adding the builder's toolkit
1248-
# would measure the build machine rather than the consumer, and a header that only compiles that way
1249-
# fails in the consumer's project instead of here.
1250-
# Headers a wheel-only consumer cannot compile and is not expected to. Each needs something outside the
1251-
# package: a platform that is not the one being built for, or a third-party library the wheel does not
1252-
# carry. They ship because a source build includes them, and holding them to this rule would report a
1253-
# defect with no available fix.
1246+
cuda_headers = (
1247+
"executorch/extension/cuda/cuda_allocator.h",
1248+
"executorch/extension/cuda/runtime_api.h",
1249+
)
1250+
cuda_includes = []
1251+
if any(
1252+
(package_dir / "lib").glob(
1253+
_library_file_name("libexecutorch_extension_cuda") + "*"
1254+
)
1255+
):
1256+
for header in cuda_headers:
1257+
assert (include_root / header).is_file(), f"{header} is not shipped"
1258+
# Only these headers require toolkit types. Use the consumer's toolkit
1259+
# search, not the compiler's default path or the builder's Torch headers.
1260+
toolkit_source = work_dir / "cuda-toolkit"
1261+
toolkit_source.mkdir(parents=True, exist_ok=True)
1262+
toolkit_build = work_dir / "cuda-toolkit-build"
1263+
(toolkit_source / "CMakeLists.txt").write_text(
1264+
"cmake_minimum_required(VERSION 3.19)\n"
1265+
"project(toolkit CXX)\n"
1266+
"find_package(CUDAToolkit REQUIRED)\n"
1267+
'file(WRITE "${CMAKE_BINARY_DIR}/includes.txt" "${CUDAToolkit_INCLUDE_DIRS}")\n'
1268+
)
1269+
command = [
1270+
_tool("cmake"),
1271+
"-S",
1272+
str(toolkit_source),
1273+
"-B",
1274+
str(toolkit_build),
1275+
]
1276+
if os.environ.get("CUDA_HOME"):
1277+
command.append(f"-DCUDAToolkit_ROOT={os.environ['CUDA_HOME']}")
1278+
subprocess.run(command, capture_output=True, text=True, check=True)
1279+
cuda_includes = [
1280+
f"-I{directory}"
1281+
for directory in (toolkit_build / "includes.txt").read_text().split(";")
1282+
if directory
1283+
]
1284+
source = work_dir / "header_probe.cpp"
12541285
needs_more_than_the_wheel = (
12551286
# These ship because other shipped headers include them, so they cannot be left out, and they do
12561287
# not compile on their own: each needs a third-party library the wheel links but publishes no
@@ -1262,7 +1293,6 @@ def test_every_shipped_header_compiles(work_dir: Path) -> None:
12621293
"torch/headeronly/util/complex_utils.h",
12631294
)
12641295

1265-
source = work_dir / "header_probe.cpp"
12661296
broken = []
12671297
skipped_but_fine = []
12681298
compiled = 0
@@ -1274,7 +1304,14 @@ def test_every_shipped_header_compiles(work_dir: Path) -> None:
12741304
f"#include <{relative.as_posix()}>\nint main() {{ return 0; }}\n"
12751305
)
12761306
result = subprocess.run(
1277-
[_cxx(), "-std=c++20", *includes, "-fsyntax-only", str(source)],
1307+
[
1308+
_cxx(),
1309+
"-std=c++20",
1310+
*includes,
1311+
*(cuda_includes if relative.as_posix() in cuda_headers else []),
1312+
"-fsyntax-only",
1313+
str(source),
1314+
],
12781315
capture_output=True,
12791316
text=True,
12801317
check=False,
@@ -1312,6 +1349,52 @@ def test_every_shipped_header_compiles(work_dir: Path) -> None:
13121349
)
13131350

13141351

1352+
def test_cuda_allocator_component(work_dir: Path) -> None:
1353+
"""The CUDA target must supply everything needed to use its allocator."""
1354+
package = _installed_package_dir()
1355+
if not any(
1356+
(package / "lib").glob(_library_file_name("libexecutorch_extension_cuda") + "*")
1357+
):
1358+
print("- no CUDA extension is shipped; skipping its allocator consumer")
1359+
return
1360+
1361+
source_dir = work_dir / "cuda-allocator-consumer"
1362+
source_dir.mkdir(parents=True, exist_ok=True)
1363+
(source_dir / "consumer.cpp").write_text(
1364+
"#include <executorch/extension/cuda/cuda_allocator.h>\n"
1365+
"using namespace executorch::extension::cuda;\n"
1366+
"int main() {\n"
1367+
" return CudaAllocator::instance().device_type() ==\n"
1368+
" executorch::runtime::etensor::DeviceType::CUDA ? 0 : 1;\n"
1369+
"}\n"
1370+
)
1371+
(source_dir / "CMakeLists.txt").write_text(_consumer_cmake(["extension_cuda"]))
1372+
build_dir = work_dir / "cuda-allocator-consumer-build"
1373+
configure = [
1374+
_tool("cmake"),
1375+
"-S",
1376+
str(source_dir),
1377+
"-B",
1378+
str(build_dir),
1379+
f"-DCMAKE_PREFIX_PATH={package}",
1380+
]
1381+
if os.environ.get("CUDA_HOME"):
1382+
configure.append(f"-DCUDAToolkit_ROOT={os.environ['CUDA_HOME']}")
1383+
for command in (configure, _cmake_build(_tool("cmake"), build_dir)):
1384+
result = subprocess.run(command, capture_output=True, text=True, check=False)
1385+
assert result.returncode == 0, (
1386+
"a consumer linking only executorch::extension_cuda could not build:\n"
1387+
f"{result.stdout[-2000:]}\n{result.stderr[-2000:]}"
1388+
)
1389+
subprocess.run(
1390+
[str(_executable(build_dir, "consumer"))],
1391+
env=_loader_clean_environment(),
1392+
check=True,
1393+
timeout=_RUN_TIMEOUT,
1394+
)
1395+
print("CUDA allocator compiles, links and runs through executorch::extension_cuda")
1396+
1397+
13151398
def test_shipped_headers_have_implementations(work_dir: Path) -> None:
13161399
"""A header that compiles but has no implementation in any shipped library is unusable.
13171400
@@ -1921,6 +2004,7 @@ def run_tests(work_dir: Path) -> None:
19212004
test_find_package_honours_a_version_request(work_dir)
19222005
test_profiler_component_is_usable(work_dir)
19232006
test_every_shipped_header_compiles(work_dir)
2007+
test_cuda_allocator_component(work_dir)
19242008
test_shipped_headers_have_implementations(work_dir)
19252009
test_documented_example_compiles(work_dir)
19262010
test_runtime_alone_links_but_cannot_compute(work_dir)

‎backends/aoti/slim/core/storage.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@
1313
#ifdef CUDA_AVAILABLE
1414
#include <executorch/backends/aoti/slim/c10/cuda/Exception.h>
1515
#include <executorch/backends/aoti/slim/cuda/guard.h>
16-
#include <executorch/backends/cuda/runtime/cuda_allocator.h>
1716
#include <executorch/extension/cuda/caller_stream.h>
17+
#include <executorch/extension/cuda/cuda_allocator.h>
1818
#endif
1919

2020
#include <executorch/backends/aoti/slim/c10/core/Device.h>

‎backends/aoti/slim/core/targets.bzl‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@ def define_common_targets():
1919
"//executorch/runtime/platform:platform",
2020
"//executorch/backends/aoti/slim/c10/cuda:exception",
2121
"//executorch/backends/aoti/slim/cuda:guard",
22-
"//executorch/backends/cuda/runtime:cuda_allocator",
2322
"//executorch/extension/cuda:caller_stream",
23+
"//executorch/extension/cuda:cuda_allocator",
2424
],
2525
)
2626

‎backends/cuda/CMakeLists.txt‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -240,7 +240,7 @@ install(
240240
)
241241

242242
# CUDA-specific AOTI shim symbols (dynamically linked)
243-
set(_aoti_cuda_shim_sources runtime/cuda_allocator.cpp runtime/shims/memory.cpp
243+
set(_aoti_cuda_shim_sources runtime/shims/memory.cpp
244244
runtime/shims/cuda_guard.cpp
245245
)
246246

@@ -353,11 +353,6 @@ set(_aoti_cuda_backend_sources
353353
if(EXECUTORCH_BUILD_EXTENSION_LLM)
354354
list(APPEND _aoti_cuda_backend_sources runtime/cuda_kv_cache.cpp)
355355
endif()
356-
if(_cuda_is_msvc_toolchain)
357-
# MSVC links aoti_cuda_backend into portable_lib without relying on C++
358-
# symbols exported from aoti_cuda_shims.dll.
359-
list(APPEND _aoti_cuda_backend_sources runtime/cuda_allocator.cpp)
360-
endif()
361356

362357
# Build the delegate as a shared library for the wheel so a process has one copy
363358
# of it, and keep it static everywhere else so no other build changes.

‎backends/cuda/runtime/cuda_allocator.h‎

Lines changed: 4 additions & 130 deletions
Original file line numberDiff line numberDiff line change
@@ -8,133 +8,7 @@
88

99
#pragma once
1010

11-
#include <executorch/extension/cuda/runtime_api.h>
12-
#include <executorch/runtime/core/device_allocator.h>
13-
14-
#include <atomic>
15-
#include <mutex>
16-
#include <unordered_map>
17-
18-
namespace executorch::backends::cuda {
19-
20-
/**
21-
* CUDA implementation of DeviceAllocator.
22-
*
23-
* Uses cudaMalloc/cudaFree for allocation and cudaMemcpy for host-device
24-
* transfers. This allocator is automatically registered as a singleton
25-
* with the DeviceAllocatorRegistry when the CUDA backend library is linked.
26-
*
27-
* All CUDA memory operations in the CUDA backend should go through this
28-
* allocator for consistent memory management.
29-
*/
30-
class CudaAllocator final : public executorch::runtime::DeviceAllocator {
31-
public:
32-
/// Alignments above 256 bytes add up to alignment minus one bytes of padding.
33-
executorch::runtime::Result<void*> allocate(
34-
size_t nbytes,
35-
executorch::runtime::etensor::DeviceIndex index,
36-
size_t alignment = kDefaultAlignment) override;
37-
38-
void deallocate(void* ptr, executorch::runtime::etensor::DeviceIndex index)
39-
override;
40-
41-
executorch::runtime::Error copy_host_to_device(
42-
void* dst,
43-
const void* src,
44-
size_t nbytes,
45-
executorch::runtime::etensor::DeviceIndex index) override;
46-
47-
executorch::runtime::Error copy_device_to_host(
48-
void* dst,
49-
const void* src,
50-
size_t nbytes,
51-
executorch::runtime::etensor::DeviceIndex index) override;
52-
53-
executorch::runtime::etensor::DeviceType device_type() const override;
54-
55-
/// Returns the global CudaAllocator singleton.
56-
static CudaAllocator& instance();
57-
58-
// --- Async (stream-based) operations for SlimTensor/Storage layer ---
59-
60-
/**
61-
* Allocate device memory asynchronously on the given CUDA stream.
62-
*/
63-
static executorch::runtime::Result<void*> allocate_async(
64-
size_t nbytes,
65-
executorch::runtime::etensor::DeviceIndex index,
66-
cudaStream_t stream);
67-
68-
/**
69-
* Deallocate device memory asynchronously on the given CUDA stream.
70-
*/
71-
static void deallocate_async(
72-
void* ptr,
73-
executorch::runtime::etensor::DeviceIndex index,
74-
cudaStream_t stream);
75-
76-
/**
77-
* Return memory this backend's device pool is holding for reuse back to the
78-
* driver.
79-
*
80-
* The pool keeps freed memory so that repeated allocations do not have to map
81-
* it again, which is what makes delegate execution cheap, so a long-lived
82-
* process should call this once its work on the device is finished. Only
83-
* frees the driver has already observed can be released, so a caller that has
84-
* not synchronized simply gets less back. Allocations that are still live are
85-
* unaffected either way.
86-
*
87-
* The pool belongs to this backend rather than being the device default pool,
88-
* so the pool trim never affects memory another user of the async allocator
89-
* is holding. The graph memory trim is the exception: it is scoped to the
90-
* device, so it also releases unused graph memory cached by other users in
91-
* this process.
92-
*
93-
* Does nothing on ROCm. HIP has equivalents for all of these calls; this
94-
* repository's CUDA-to-HIP compatibility header does not alias them yet.
95-
*
96-
* @param index Device to release on, or a negative value to release every
97-
* device this backend has allocated on. Note that means every device, not
98-
* the current one, which is what a negative value means elsewhere in this
99-
* class: a delegate is often torn down from a thread that is not current
100-
* on the device it ran on, so releasing only the current device would leave
101-
* that memory held.
102-
*/
103-
static void release_cached_memory(
104-
executorch::runtime::etensor::DeviceIndex index);
105-
106-
#if !defined(EXECUTORCH_USE_HIP)
107-
/**
108-
* The memory pool this backend allocates from on a device, or nullptr if it
109-
* has not allocated there or the pool could not be created.
110-
*
111-
* Exposed so a test can observe what the pool is holding, which is not
112-
* visible through the device default pool. No production caller.
113-
*
114-
* Not declared on ROCm: the pool code is compiled out there, so there is
115-
* nothing to observe and the pool type needs no HIP alias.
116-
*
117-
* @param index Device to query, or a negative value for the current one.
118-
*/
119-
static cudaMemPool_t pool_for_device(
120-
executorch::runtime::etensor::DeviceIndex index);
121-
#endif // !EXECUTORCH_USE_HIP
122-
123-
/**
124-
* Copy memory asynchronously on the given CUDA stream.
125-
* Supports H2D, D2H, and D2D based on src/dst device types.
126-
*/
127-
static executorch::runtime::Error memcpy_async(
128-
void* dst,
129-
const void* src,
130-
size_t nbytes,
131-
cudaMemcpyKind direction,
132-
cudaStream_t stream);
133-
134-
private:
135-
std::mutex padded_allocations_mutex_;
136-
std::unordered_map<void*, void*> padded_allocations_;
137-
std::atomic<size_t> padded_allocation_count_{0};
138-
};
139-
140-
} // namespace executorch::backends::cuda
11+
// This header is a stub left behind after the move to
12+
// executorch/extension/cuda. As such, it is deprecated; include and
13+
// use the below header directly instead.
14+
#include <executorch/extension/cuda/cuda_allocator.h>

‎backends/cuda/runtime/cuda_backend.cpp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,14 @@
4343
// Include our shim layer headers
4444
#include <executorch/backends/aoti/aoti_delegate_handle.h>
4545
#include <executorch/backends/aoti/utils.h>
46-
#include <executorch/backends/cuda/runtime/cuda_allocator.h>
4746
#include <executorch/backends/cuda/runtime/cuda_delegate_handle.h>
4847
#include <executorch/backends/cuda/runtime/cuda_kv_cache.h>
4948
#include <executorch/backends/cuda/runtime/cuda_mutable_state.h>
5049
#include <executorch/backends/cuda/runtime/cuda_weight_cache.h>
5150
#include <executorch/backends/cuda/runtime/platform/platform.h>
5251
#include <executorch/backends/cuda/runtime/shims/memory.h>
5352
#include <executorch/backends/cuda/runtime/utils.h>
53+
#include <executorch/extension/cuda/cuda_allocator.h>
5454
#include <executorch/extension/llm/cache/cache_registry.h>
5555

5656
namespace executorch::backends::cuda {
@@ -1128,6 +1128,7 @@ class ET_EXPERIMENTAL CudaBackend final
11281128
cudaGetErrorString(sync_err));
11291129
(void)cudaGetLastError();
11301130
}
1131+
// This also releases unused blocks cached by other allocator users.
11311132
CudaAllocator::release_cached_memory(-1);
11321133
}
11331134
}

‎backends/cuda/runtime/cuda_kv_cache.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
#include <executorch/backends/aoti/slim/c10/core/ScalarType.h>
2323
#include <executorch/backends/aoti/slim/core/slim_tensor.h>
2424
#include <executorch/backends/aoti/slim/factory/from_blob.h>
25-
#include <executorch/backends/cuda/runtime/cuda_allocator.h>
25+
#include <executorch/extension/cuda/cuda_allocator.h>
2626
#include <executorch/extension/llm/cache/cache_registry.h>
2727
#include <executorch/extension/llm/cache/sequence_cache.h>
2828
#include <executorch/runtime/platform/log.h>

0 commit comments

Comments
 (0)