Skip to content

Commit 0f2ae90

Browse files
authored
Document both shapes for device resident outputs (pytorch#23102)
The CUDA page tells you to export device resident inputs and outputs with unplanned graph outputs. A Python caller cannot follow that recipe. An unplanned output means the program reserves no buffer, so the caller has to supply one before the method runs. From C++ that is natural, since the caller makes a tensor on the device and hands over its pointer. The Python bindings do it for the caller and they do it on the host, so a device resident output is given host memory and the run fails. The setting the page warns against is the one that works from Python. With `enable_non_cpu_memory_planning` on, a planned output buffer is allocated on the device, and `skip_d2h_for_method_outputs` means nothing copies it back, so the result stays on the device and the caller receives a pointer to the runtime's own buffer. That buffer belongs to the method and is reused by the next execution, which the page now says. It is also what the exporters in this repository already do, for example `examples/models/gemma4_31b/export.py`. The input half of the recipe was correct and is unchanged. The output half now describes two shapes: planned when the caller cannot supply device memory, which is every Python caller today, and unplanned when it can, which today means a C++ caller handing the runtime a device tensor through `Module::set_output`. Fixes pytorch#23101 Test plan: read against the example exporters in this repository, which use unplanned inputs and planned outputs together with both copy skips. No code changed.
1 parent fc21064 commit 0f2ae90

1 file changed

Lines changed: 26 additions & 12 deletions

File tree

‎docs/source/backends/cuda/cuda-overview.md‎

Lines changed: 26 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -140,13 +140,13 @@ exec_program = et_program.to_executorch(
140140
skip_d2h_for_method_outputs=True,
141141
),
142142
enable_non_cpu_memory_planning=True,
143-
# Required alongside the skips. Memory planning reserves a buffer for graph inputs and
144-
# outputs by default, and the runtime fills a planned input by copying the caller's
145-
# memory into that buffer, which reintroduces the copy you just asked to skip. On device
146-
# memory that copy is a host memcpy into a device pointer, which is undefined.
147-
memory_planning_pass=MemoryPlanningPass(
148-
alloc_graph_input=False, alloc_graph_output=False
149-
),
143+
# Required alongside the skips. Memory planning reserves a buffer for graph inputs by
144+
# default, and the runtime fills a planned input by copying the caller's memory into that
145+
# buffer, which reintroduces the copy you just asked to skip. On device memory that copy is
146+
# a host memcpy into a device pointer, which is undefined.
147+
# Outputs stay planned, so the runtime allocates them on the device and hands back a
148+
# pointer to its own buffer. See the note below for when to unplan them instead.
149+
memory_planning_pass=MemoryPlanningPass(alloc_graph_input=False),
150150
)
151151
)
152152
```
@@ -169,11 +169,25 @@ out works, and the snippet above sets it explicitly only to make the requirement
169169
it to `False` while asking for either skip raises a `ValueError`, because copy insertion happens
170170
during device-aware memory planning.
171171

172-
**Both flags also need unplanned graph inputs and outputs**, via
173-
`MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)` as shown above. Without
174-
it the program still reserves its own buffer and the runtime copies into it, so the copy comes
175-
back at run time. On device memory that copy is a host memcpy into a device pointer, which is
176-
undefined, so the program crashes rather than returning a wrong answer.
172+
**Inputs must be unplanned**, via `MemoryPlanningPass(alloc_graph_input=False)`. Without it the
173+
program reserves its own input buffer and the runtime copies the caller's memory into it, so the
174+
copy comes back at run time. On device memory that copy is a host memcpy into a device pointer,
175+
which is undefined, so the program crashes rather than returning a wrong answer.
176+
177+
**Outputs have two valid shapes, and which one you want depends on the language.**
178+
179+
Leave outputs planned, which is the default, when the caller cannot supply device memory itself.
180+
The planned buffer is allocated on the device because `enable_non_cpu_memory_planning` is on, and
181+
`skip_d2h_for_method_outputs` means nothing copies it back, so the result stays on the device and
182+
the caller receives a pointer to the runtime's own buffer. That buffer belongs to the method and is
183+
reused by the next execution, so read or copy the result before running again. This is the shape
184+
every Python caller wants, and it is what the exporters in this repository use, for example
185+
`examples/models/gemma4_31b/export.py`.
186+
187+
Set `alloc_graph_output=False` only when the caller supplies the output memory itself, which today
188+
means a C++ caller handing the runtime a device tensor through `Module::set_output`. The Python
189+
bindings allocate host memory for an unplanned output, so a device resident method exported this
190+
way cannot run from Python.
177191

178192
**Per-method selection is on the outer config, not on the flags.** Pass a dict of
179193
`PropagateDeviceConfig` keyed by method name:

0 commit comments

Comments
 (0)