Skip to content

Commit 1b2838b

Browse files
authored
Document how to hand GPU memory straight to a CUDA model (pytorch#21577)
A CUDA model has to get its input data onto the GPU somehow, and there are two ways. You pick one at export time. **By default the runtime copies for you.** Export inserts a host-to-device copy in front of every delegate input and a device-to-host copy after every output, so you pass ordinary CPU tensors: ```python outputs = method([torch.randn(4, 64)]) ``` **If your data is already on the GPU, those copies are wasted.** You can turn them off, and then the input placeholders are treated as device memory: ```python exec_program = et_program.to_executorch( ExecutorchBackendConfig( propagate_device_config=PropagateDeviceConfig( skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True, ), enable_non_cpu_memory_planning=True, memory_planning_pass=MemoryPlanningPass( alloc_graph_input=False, alloc_graph_output=False ), ) ) ``` ```python outputs = method([torch.randn(4, 64, device="cuda")]) ``` Outputs stay on the device too, which is what you want when the next stage of the pipeline is also on the GPU. ## Why this PR None of this appears anywhere in the documentation. I only found it by reading `PropagateDevicePass`, and I got it wrong twice on the way. The first mistake was assuming the two skip flags were enough. They are not. Memory planning reserves a buffer for every graph input and output by default, and the runtime fills a planned input by copying the caller's memory into that buffer, which puts back the copy the export just asked to skip. On device memory that copy is a host `memcpy` into a GPU pointer, which is undefined. So the pairing with `MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)` is required, not optional, and the page now says so where the recipe is given. The second mistake was not realising the choice is baked into the `.pte`. A program exported with copies expects host tensors and one exported without them expects device tensors, so the two are not interchangeable at run time. The new section covers both modes, the memory-planning requirement, the fact that `enable_non_cpu_memory_planning` already defaults to `True`, how to select per method, and the matching C++ side where the input tensor is built over a device pointer. Documentation only, no code changes. ## A note on selecting per method Per-method selection works on `propagate_device_config` itself, which accepts a dict keyed by method name: ```python ExecutorchBackendConfig( propagate_device_config={ "forward": PropagateDeviceConfig( skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True ), "forward_from_host": PropagateDeviceConfig(), }, ... ) ``` It does not work by passing a dict to `skip_h2d_for_method_inputs`. That field is read for truthiness, so any non-empty dict enables the skip for every method whatever the values inside it say. The page warns against that form rather than showing it. ## Tested Exercised both modes on real hardware while writing this, on x86_64 and on two Jetson devices. The default export contained two copy operators and the opt-out export contained none, which is the difference the flags control. Checked the claims the page makes rather than assuming them: - exported a two-input model with the recipe above and confirmed memory planning leaves the graph inputs and outputs unallocated, where the earlier version of this page left them allocated and would have hit the undefined copy - `enable_non_cpu_memory_planning` is declared `bool = True`, so omitting it raises nothing, and the guard that raises `ValueError` fires only when either skip is set alongside an explicit `False` - `skip_h2d_for_method_inputs` is declared `bool` and read for truthiness, so a non-empty dict enables the skip whatever it contains, which is why the page steers to the outer `propagate_device_config` form that resolves per method by name Every Python snippet in the page parses.
1 parent 3d8258b commit 1b2838b

1 file changed

Lines changed: 120 additions & 0 deletions

File tree

‎docs/source/backends/cuda/cuda-overview.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,126 @@ No additional steps are necessary to use the backend beyond linking the target.
103103

104104
----
105105

106+
## Activation memory: who owns the GPU copies
107+
108+
A CUDA model has to get its input data onto the GPU somehow. There are two ways, and you pick
109+
one at export time.
110+
111+
**Default: the runtime copies for you.** Export inserts a host-to-device copy in front of each
112+
delegate input and a device-to-host copy after each delegate output. You pass ordinary CPU
113+
tensors and the runtime moves the data:
114+
115+
```python
116+
exec_program = et_program.to_executorch()
117+
```
118+
119+
```python
120+
# CPU tensors. The runtime copies them to the GPU and copies results back.
121+
outputs = method([torch.randn(4, 64)])
122+
```
123+
124+
This is the simplest option and the right default. The cost is a copy in each direction on
125+
every call.
126+
127+
**Opt out: you hand over GPU memory directly.** If your data is already on the device, those
128+
copies are wasted. Turn them off and the input placeholders are treated as device memory
129+
instead:
130+
131+
```python
132+
from executorch.exir import ExecutorchBackendConfig
133+
from executorch.exir.passes import MemoryPlanningPass
134+
from executorch.exir.passes.propagate_device_config import PropagateDeviceConfig
135+
136+
exec_program = et_program.to_executorch(
137+
ExecutorchBackendConfig(
138+
propagate_device_config=PropagateDeviceConfig(
139+
skip_h2d_for_method_inputs=True,
140+
skip_d2h_for_method_outputs=True,
141+
),
142+
enable_non_cpu_memory_planning=True,
143+
# Required alongside the skips. Memory planning reserves a buffer for graph inputs and
144+
# outputs by default, and the runtime fills a planned input by copying the caller's
145+
# memory into that buffer, which reintroduces the copy you just asked to skip. On device
146+
# memory that copy is a host memcpy into a device pointer, which is undefined.
147+
memory_planning_pass=MemoryPlanningPass(
148+
alloc_graph_input=False, alloc_graph_output=False
149+
),
150+
)
151+
)
152+
```
153+
154+
```python
155+
# Now the tensors must already be on the GPU. Passing CPU tensors is a caller error.
156+
outputs = method([torch.randn(4, 64, device="cuda")])
157+
```
158+
159+
Outputs also stay on the device, which is what you want when the next stage of your pipeline is
160+
also on the GPU.
161+
162+
### Things worth knowing
163+
164+
**The two flags are independent.** Skip only the input copies if you produce data on the GPU but
165+
want results back on the host, or only the output copies for the reverse.
166+
167+
**Both flags need `enable_non_cpu_memory_planning=True`.** It defaults to `True`, so leaving it
168+
out works, and the snippet above sets it explicitly only to make the requirement visible. Setting
169+
it to `False` while asking for either skip raises a `ValueError`, because copy insertion happens
170+
during device-aware memory planning.
171+
172+
**Both flags also need unplanned graph inputs and outputs**, via
173+
`MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)` as shown above. Without
174+
it the program still reserves its own buffer and the runtime copies into it, so the copy comes
175+
back at run time. On device memory that copy is a host memcpy into a device pointer, which is
176+
undefined, so the program crashes rather than returning a wrong answer.
177+
178+
**Per-method selection is on the outer config, not on the flags.** Pass a dict of
179+
`PropagateDeviceConfig` keyed by method name:
180+
181+
```python
182+
ExecutorchBackendConfig(
183+
propagate_device_config={
184+
"forward": PropagateDeviceConfig(
185+
skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True
186+
),
187+
"forward_from_host": PropagateDeviceConfig(),
188+
},
189+
...
190+
)
191+
```
192+
193+
Do not pass a dict to `skip_h2d_for_method_inputs` itself. The pass reads that field for
194+
truthiness, so any non-empty dict enables the skip for every method regardless of the values
195+
inside it.
196+
197+
**The choice is baked into the `.pte`.** A program exported with copies expects host tensors; one
198+
exported without them expects device tensors. Passing the wrong kind is a caller error, not
199+
something the runtime corrects, so the two are not interchangeable at run time.
200+
201+
**From C++ it is the same contract.** With the copies skipped, the input has to be a tensor
202+
the delegate will accept as device resident. `clone_tensor_ptr_to` allocates on the device
203+
and copies the data across:
204+
205+
```cpp
206+
auto host_input = make_tensor_ptr({rows, columns}, std::move(data));
207+
auto input = clone_tensor_ptr_to(host_input, DeviceType::CUDA);
208+
auto result = module.forward(input);
209+
```
210+
211+
`from_blob` is not enough on its own. It has no device parameter, so the tensor it returns
212+
is tagged CPU whatever the pointer points at, and the delegate rejects it.
213+
214+
**How the delegate decides a tensor is device resident.** Two checks, in this order. First
215+
the tensor's own `device_type` has to be `CUDA`, which is metadata set by whoever built the
216+
tensor. Then, because that tag is only a claim, the delegate calls
217+
`cudaPointerGetAttributes` on the data pointer and requires the memory to really be
218+
`cudaMemoryTypeDevice` or `cudaMemoryTypeManaged`. A CUDA-tagged tensor backed by host
219+
memory is caught there rather than corrupting the run.
220+
221+
With the default export, pass an ordinary host tensor instead and the runtime handles the
222+
transfer.
223+
224+
----
225+
106226
## Examples
107227
108228
For complete end-to-end examples of exporting and running models with the CUDA backend, see:

0 commit comments

Comments
 (0)