Repository navigation
Commit 3d8258b
authored
Document and test the memory planning a device-activation export needs (pytorch#21703)
## The problem
ExecuTorch can export a model so its activations stay on the GPU. You
hand over memory
that is already on the GPU, the model runs on it, and the output stays
there. No copying
back and forth, which is the point of the feature.
Two settings control this, and they live in different places. The first
says "do not
insert copies at the delegate boundary":
```python
PropagateDeviceConfig(
skip_h2d_for_method_inputs=True,
skip_d2h_for_method_outputs=True,
)
```
The second is memory planning, which by default reserves a buffer for
every graph input
and output. That is the right default for an ordinary model, and wrong
here: the runtime
reserves its own GPU buffer for an input you already allocated, and then
fills it by
copying your memory into it. The copy the export asked to skip comes
back at run time.
It is worse than a wasted copy. For device memory that copy is a plain
host `memcpy`
into a GPU pointer, which is undefined and crashes the process with no
message.
Nothing said the two settings had to be paired, and no test exported
such a program and
checked it. The result is a documented feature whose documented usage
does not work.
## The change
Documentation and a test. No behavior change.
The configuration now states the pairing, with a complete example:
```python
ExecutorchBackendConfig(
propagate_device_config=PropagateDeviceConfig(
skip_h2d_for_method_inputs=True,
skip_d2h_for_method_outputs=True,
),
enable_non_cpu_memory_planning=True,
memory_planning_pass=MemoryPlanningPass(
alloc_graph_input=False, alloc_graph_output=False
),
)
```
The measurable difference, for a model with two small inputs:
```
graph inputs allocated (default): 768 planned bytes, the runtime copies into its own buffer
graph inputs not allocated: 256 planned bytes, the runtime points at the caller's memory
```
The extra 512 bytes are a duplicate of the caller's two inputs.
## Test plan
Added `test_skipping_copies_requires_unallocated_graph_io`, which pins
the pairing:
with default planning the graph inputs are planned, and with the pairing
none of the
inputs or outputs are. It needs no GPU, because it inspects the exported
program rather
than running it.
Confirmed the test fails without the pairing:
```
AssertionError: True is not false : no graph input should be planned
when the caller provides it, got [True, True]
```
Ran the full `test_propagate_device_pass` file, 24 tests, all passing.
Separately verified on hardware that a program exported with the pairing
runs correctly
from C++ with GPU memory in and out, matching eager PyTorch exactly on
two different GPU
architectures.1 parent b8b02f8 commit 3d8258b
2 files changed
Lines changed: 96 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
28 | 49 | | |
29 | 50 | | |
30 | 51 | | |
31 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
32 | 56 | | |
33 | 57 | | |
34 | 58 | | |
35 | 59 | | |
36 | 60 | | |
37 | 61 | | |
38 | 62 | | |
| 63 | + | |
| 64 | + | |
39 | 65 | | |
40 | 66 | | |
41 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
1270 | 1271 | | |
1271 | 1272 | | |
1272 | 1273 | | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
1273 | 1343 | | |
1274 | 1344 | | |
1275 | 1345 | | |
0 commit comments