Repository navigation
Commit 1b2838b
authored
Document how to hand GPU memory straight to a CUDA model (pytorch#21577)
A CUDA model has to get its input data onto the GPU somehow, and there
are two ways. You
pick one at export time.
**By default the runtime copies for you.** Export inserts a
host-to-device copy in front of
every delegate input and a device-to-host copy after every output, so
you pass ordinary CPU
tensors:
```python
outputs = method([torch.randn(4, 64)])
```
**If your data is already on the GPU, those copies are wasted.** You can
turn them off, and
then the input placeholders are treated as device memory:
```python
exec_program = et_program.to_executorch(
ExecutorchBackendConfig(
propagate_device_config=PropagateDeviceConfig(
skip_h2d_for_method_inputs=True,
skip_d2h_for_method_outputs=True,
),
enable_non_cpu_memory_planning=True,
memory_planning_pass=MemoryPlanningPass(
alloc_graph_input=False, alloc_graph_output=False
),
)
)
```
```python
outputs = method([torch.randn(4, 64, device="cuda")])
```
Outputs stay on the device too, which is what you want when the next
stage of the pipeline
is also on the GPU.
## Why this PR
None of this appears anywhere in the documentation. I only found it by
reading
`PropagateDevicePass`, and I got it wrong twice on the way.
The first mistake was assuming the two skip flags were enough. They are
not. Memory
planning reserves a buffer for every graph input and output by default,
and the runtime
fills a planned input by copying the caller's memory into that buffer,
which puts back the
copy the export just asked to skip. On device memory that copy is a host
`memcpy` into a
GPU pointer, which is undefined. So the pairing with
`MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)`
is required, not
optional, and the page now says so where the recipe is given.
The second mistake was not realising the choice is baked into the
`.pte`. A program
exported with copies expects host tensors and one exported without them
expects device
tensors, so the two are not interchangeable at run time.
The new section covers both modes, the memory-planning requirement, the
fact that
`enable_non_cpu_memory_planning` already defaults to `True`, how to
select per method, and
the matching C++ side where the input tensor is built over a device
pointer.
Documentation only, no code changes.
## A note on selecting per method
Per-method selection works on `propagate_device_config` itself, which
accepts a dict keyed
by method name:
```python
ExecutorchBackendConfig(
propagate_device_config={
"forward": PropagateDeviceConfig(
skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True
),
"forward_from_host": PropagateDeviceConfig(),
},
...
)
```
It does not work by passing a dict to `skip_h2d_for_method_inputs`. That
field is read for
truthiness, so any non-empty dict enables the skip for every method
whatever the values
inside it say. The page warns against that form rather than showing it.
## Tested
Exercised both modes on real hardware while writing this, on x86_64 and
on two Jetson
devices. The default export contained two copy operators and the opt-out
export contained
none, which is the difference the flags control.
Checked the claims the page makes rather than assuming them:
- exported a two-input model with the recipe above and confirmed memory
planning leaves
the graph inputs and outputs unallocated, where the earlier version of
this page left
them allocated and would have hit the undefined copy
- `enable_non_cpu_memory_planning` is declared `bool = True`, so
omitting it raises
nothing, and the guard that raises `ValueError` fires only when either
skip is set
alongside an explicit `False`
- `skip_h2d_for_method_inputs` is declared `bool` and read for
truthiness, so a non-empty
dict enables the skip whatever it contains, which is why the page steers
to the outer
`propagate_device_config` form that resolves per method by name
Every Python snippet in the page parses.1 parent 3d8258b commit 1b2838b
1 file changed
Lines changed: 120 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
106 | 226 | | |
107 | 227 | | |
108 | 228 | | |
| |||
0 commit comments