Repository navigation
Commit 0f2ae90
authored
Document both shapes for device resident outputs (pytorch#23102)
The CUDA page tells you to export device resident inputs and outputs
with unplanned graph outputs.
A Python caller cannot follow that recipe.
An unplanned output means the program reserves no buffer, so the caller
has to supply one before the
method runs. From C++ that is natural, since the caller makes a tensor
on the device and hands over
its pointer. The Python bindings do it for the caller and they do it on
the host, so a device
resident output is given host memory and the run fails.
The setting the page warns against is the one that works from Python.
With
`enable_non_cpu_memory_planning` on, a planned output buffer is
allocated on the device, and
`skip_d2h_for_method_outputs` means nothing copies it back, so the
result stays on the device and
the caller receives a pointer to the runtime's own buffer. That buffer
belongs to the method and is
reused by the next execution, which the page now says. It is also what
the exporters in this
repository already do, for example
`examples/models/gemma4_31b/export.py`.
The input half of the recipe was correct and is unchanged. The output
half now describes two shapes:
planned when the caller cannot supply device memory, which is every
Python caller today, and
unplanned when it can, which today means a C++ caller handing the
runtime a device tensor through
`Module::set_output`.
Fixes pytorch#23101
Test plan: read against the example exporters in this repository, which
use unplanned inputs and
planned outputs together with both copy skips. No code changed.1 parent fc21064 commit 0f2ae90
1 file changed
Lines changed: 26 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
177 | 191 | | |
178 | 192 | | |
179 | 193 | | |
| |||
0 commit comments