Skip to content

Commit 3d8258b

Browse files
authored
Document and test the memory planning a device-activation export needs (pytorch#21703)
## The problem ExecuTorch can export a model so its activations stay on the GPU. You hand over memory that is already on the GPU, the model runs on it, and the output stays there. No copying back and forth, which is the point of the feature. Two settings control this, and they live in different places. The first says "do not insert copies at the delegate boundary": ```python PropagateDeviceConfig( skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True, ) ``` The second is memory planning, which by default reserves a buffer for every graph input and output. That is the right default for an ordinary model, and wrong here: the runtime reserves its own GPU buffer for an input you already allocated, and then fills it by copying your memory into it. The copy the export asked to skip comes back at run time. It is worse than a wasted copy. For device memory that copy is a plain host `memcpy` into a GPU pointer, which is undefined and crashes the process with no message. Nothing said the two settings had to be paired, and no test exported such a program and checked it. The result is a documented feature whose documented usage does not work. ## The change Documentation and a test. No behavior change. The configuration now states the pairing, with a complete example: ```python ExecutorchBackendConfig( propagate_device_config=PropagateDeviceConfig( skip_h2d_for_method_inputs=True, skip_d2h_for_method_outputs=True, ), enable_non_cpu_memory_planning=True, memory_planning_pass=MemoryPlanningPass( alloc_graph_input=False, alloc_graph_output=False ), ) ``` The measurable difference, for a model with two small inputs: ``` graph inputs allocated (default): 768 planned bytes, the runtime copies into its own buffer graph inputs not allocated: 256 planned bytes, the runtime points at the caller's memory ``` The extra 512 bytes are a duplicate of the caller's two inputs. ## Test plan Added `test_skipping_copies_requires_unallocated_graph_io`, which pins the pairing: with default planning the graph inputs are planned, and with the pairing none of the inputs or outputs are. It needs no GPU, because it inspects the exported program rather than running it. Confirmed the test fails without the pairing: ``` AssertionError: True is not false : no graph input should be planned when the caller provides it, got [True, True] ``` Ran the full `test_propagate_device_pass` file, 24 tests, all passing. Separately verified on hardware that a program exported with the pairing runs correctly from C++ with GPU memory in and out, matching eager PyTorch exactly on two different GPU architectures.
1 parent b8b02f8 commit 3d8258b

2 files changed

Lines changed: 96 additions & 0 deletions

File tree

‎exir/passes/propagate_device_config.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,17 +25,43 @@
2525
@compatibility(is_backward_compatible=False)
2626
@dataclass
2727
class PropagateDeviceConfig:
28+
"""Controls whether the runtime copies method inputs and outputs across the device boundary.
29+
30+
Skipping a copy also means the runtime must not reserve its own buffer for the tensor, or it
31+
would fill that buffer from the caller's memory and the copy would come back. Memory planning
32+
allocates graph inputs and outputs by default, so a program using either skip below needs::
33+
34+
ExecutorchBackendConfig(
35+
propagate_device_config=PropagateDeviceConfig(
36+
skip_h2d_for_method_inputs=True,
37+
skip_d2h_for_method_outputs=True,
38+
),
39+
enable_non_cpu_memory_planning=True,
40+
memory_planning_pass=MemoryPlanningPass(
41+
alloc_graph_input=False, alloc_graph_output=False
42+
),
43+
)
44+
45+
Without ``alloc_graph_input=False`` the runtime reserves device memory for an input the caller
46+
already owns, then copies into it with a host memcpy, which is undefined for device memory.
47+
"""
48+
2849
# When True, method-level input tensors that feed directly into a device
2950
# delegate are NOT wrapped with _h2d_copy. The user must provide tensors
3051
# already on the target device. Useful for pipelines where inputs are
3152
# pre-staged on GPU.
53+
#
54+
# Pair with MemoryPlanningPass(alloc_graph_input=False), or the runtime reserves its own buffer
55+
# for the input and copies the caller's memory into it, which is what this exists to avoid.
3256
# A dict can be used to set per-method values, keyed by method name.
3357
skip_h2d_for_method_inputs: Union[bool, Dict[str, bool]] = False
3458

3559
# When True, device delegate outputs that are directly method outputs
3660
# are NOT wrapped with _d2h_copy. The method outputs stay on device.
3761
# Useful for cross-method GPU pipelines where the next method consumes
3862
# GPU tensors directly.
63+
#
64+
# Pair with MemoryPlanningPass(alloc_graph_output=False) for the same reason as the input flag.
3965
# A dict can be used to set per-method values, keyed by method name.
4066
skip_d2h_for_method_outputs: Union[bool, Dict[str, bool]] = False
4167

‎exir/tests/test_propagate_device_pass.py‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
from executorch.exir.capture._config import ExecutorchBackendConfig
3030
from executorch.exir.delegate import executorch_call_delegate
3131
from executorch.exir.dialects._ops import ops as exir_ops
32+
from executorch.exir.passes import MemoryPlanningPass
3233
from executorch.exir.passes.propagate_device_pass import (
3334
_get_target_device_from_compile_specs,
3435
_parse_device_spec_value,
@@ -1270,6 +1271,75 @@ def test_tensorspec_repr_includes_device(self):
12701271
self.assertIn("device=", repr_str)
12711272
self.assertIn("CPU", repr_str)
12721273

1274+
def test_skipping_copies_requires_unallocated_graph_io(self):
1275+
"""Skipping a boundary copy is only complete when the runtime also stops reserving its own
1276+
buffer for that tensor.
1277+
1278+
Memory planning allocates graph inputs and outputs by default. A planned input carries
1279+
allocation info, and the runtime fills a planned input by copying the caller's memory into the
1280+
buffer it reserved, so the copy the export asked to skip comes back at run time. For device
1281+
memory that copy is also a host memcpy into a device pointer, which crashes.
1282+
1283+
This pins the pairing the configuration needs, because the two settings live in different
1284+
places and nothing else connects them.
1285+
"""
1286+
1287+
class Model(torch.nn.Module):
1288+
def forward(self, a, b):
1289+
return torch.add(a, b)
1290+
1291+
model = Model()
1292+
inputs = (torch.randn(2, 2), torch.randn(2, 2))
1293+
skip_copies = PropagateDeviceConfig(
1294+
skip_h2d_for_method_inputs=True,
1295+
skip_d2h_for_method_outputs=True,
1296+
)
1297+
1298+
def planned_io(config: ExecutorchBackendConfig):
1299+
"""Whether the program reserves its own buffer for each graph input and output."""
1300+
lowered = to_edge_transform_and_lower(
1301+
torch.export.export(model, inputs),
1302+
partitioner=[DeviceAwarePartitioner("cuda:0")],
1303+
).to_executorch(config)
1304+
plan = lowered.executorch_program.execution_plan[0]
1305+
planned = lambda indices: [ # noqa: E731
1306+
getattr(plan.values[index].val, "allocation_info", None) is not None
1307+
for index in indices
1308+
]
1309+
return planned(plan.inputs), planned(plan.outputs)
1310+
1311+
# Default planning reserves buffers, so the runtime copies into them despite the skip flags.
1312+
planned_inputs, planned_outputs = planned_io(
1313+
ExecutorchBackendConfig(
1314+
propagate_device_config=skip_copies,
1315+
enable_non_cpu_memory_planning=True,
1316+
)
1317+
)
1318+
self.assertTrue(
1319+
all(planned_inputs),
1320+
"graph inputs are expected to be planned by default, which is why the pairing below "
1321+
f"is needed, but got {planned_inputs}",
1322+
)
1323+
1324+
# Asking planning to leave them alone is what actually removes the copy.
1325+
planned_inputs, planned_outputs = planned_io(
1326+
ExecutorchBackendConfig(
1327+
propagate_device_config=skip_copies,
1328+
enable_non_cpu_memory_planning=True,
1329+
memory_planning_pass=MemoryPlanningPass(
1330+
alloc_graph_input=False, alloc_graph_output=False
1331+
),
1332+
)
1333+
)
1334+
self.assertFalse(
1335+
any(planned_inputs),
1336+
f"no graph input should be planned when the caller provides it, got {planned_inputs}",
1337+
)
1338+
self.assertFalse(
1339+
any(planned_outputs),
1340+
f"no graph output should be planned when it stays on the device, got {planned_outputs}",
1341+
)
1342+
12731343

12741344
if __name__ == "__main__":
12751345
unittest.main()

0 commit comments

Comments
 (0)