Skip to content

Commit 187442b

Browse files
authored
Record the input tensor's device when converting it for a method (pytorch#21764)
When a tensor is passed to a method, the bindings convert it into the runtime's own tensor type. That conversion copied the shape, the strides, the dim order and the data type, and left the device at CPU regardless of what the caller passed. A buffer on an accelerator was therefore described as host memory. Both entry points are fixed, because both did this. One is reached by loading a module and calling a method on it, the other by loading a program and setting a method's inputs directly. They had the same defect and only one of them would have been worth fixing alone. This records the device instead. A device the runtime has no type for is reported by name, naming the two it does have, rather than being silently relabelled as CPU. An empty tensor is left alone, because it has no buffer for a label to describe and rejecting it would break a case that worked before. One behaviour change worth naming: a non-empty tensor on any device other than CPU or CUDA is now rejected where it was previously accepted and relabelled CPU. That includes a device whose memory the host can read, such as one a private backend registers, which would have happened to work. Nothing here passes such a tensor, and a silently wrong label is worse than a named error, but the case is real and the error says which device it saw. What this does not do: for a CPU input the conversion is unchanged, so nothing about that path is fixed here. And the label has no reader yet, so passing memory the host cannot address still misbehaves downstream, because sharing a buffer between tensors of this type does not compare devices the way sharing between the other tensor type does, and copying compares them on neither side. Making the label meaningful means adding those comparisons, which is a separate change. This one stops the conversion from discarding what the caller said. Depends on the change that adds the device mapping, since there was previously no shared way to express this and each site would have carried its own copy of the table. Test Plan: extension/pybindings/test/test_pybindings.py gains a case per entry point, each asserting the device is named as Python spells it, so an uppercased or index-less name would not match. test_method_rejects_input_on_unrepresentable_device test_program_method_rejects_input_on_unrepresentable_device test_method_accepts_a_cpu_input_after_the_device_check The first two are skipped under the ATen build, where the runtime uses the PyTorch tensor type directly and no conversion happens. The third runs in both. What these do not cover, stated plainly: none of them observes the recorded device. The bindings expose no way to read a tensor's device from Python, so a change that dropped the device again, or that stopped handling CUDA specifically, would leave this file green. The tests here pin the error path and the CPU path. The device plumbing itself is covered by the C++ tests in the two changes below this one, which can read the label directly. Built a wheel from this change, installed it in a clean environment, and ran all three: they pass. Against the code before this change the program one aborts the process rather than raising, so that path was genuinely unprotected. Also exported and ran three programs through the bindings, one using portable kernels, one using quantized operators and one using XNNPACK, to confirm the ordinary path is unaffected. One unrelated fix carried along because it sits on the lines being edited: a missing space in the neighbouring contiguity error, which printed "Input 0for method" at both sites.
1 parent 02ad1e4 commit 187442b

2 files changed

Lines changed: 90 additions & 3 deletions

File tree

‎extension/pybindings/pybindings.cpp‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ using torch::executor::ETDumpGen;
103103
#ifndef USE_ATEN_LIB
104104
using ::executorch::extension::alias_attensor_to_etensor;
105105
using ::executorch::extension::alias_etensor_to_attensor;
106+
using ::executorch::extension::torch_to_executorch_device;
106107
using ::executorch::extension::torch_to_executorch_scalar_type;
107108
#endif // !USE_ATEN_LIB
108109

@@ -794,18 +795,35 @@ struct PyModule final {
794795
at_tensor.dim() == 4) {
795796
dim_order = decltype(dim_order)({0, 2, 3, 1});
796797
} else {
797-
auto error_msg = "Input " + std::to_string(i) + "for method " +
798+
auto error_msg = "Input " + std::to_string(i) + " for method " +
798799
method_name + " should be contiguous or channels-last.";
799800
throw std::runtime_error(error_msg);
800801
}
801802
input_dim_order.push_back(std::move(dim_order));
803+
// The runtime has two device types, so a device outside that pair
804+
// cannot be represented at all and the tensor would carry a label that
805+
// does not describe its memory.
806+
auto mapped_device = torch_to_executorch_device(at_tensor.device());
807+
// An empty tensor has no buffer for a label to describe, so it is left
808+
// alone rather than rejected for a device it never touches.
809+
if (!mapped_device.has_value() && at_tensor.numel() != 0) {
810+
throw std::runtime_error(
811+
"Input " + std::to_string(i) + " for method " + method_name +
812+
" is on device " + at_tensor.device().str() +
813+
", and only CPU and CUDA tensors can be passed to a method.");
814+
}
815+
const auto device = mapped_device.value_or(
816+
torch::executor::Device(torch::executor::DeviceType::CPU));
802817
input_tensors.emplace_back(
803818
type,
804819
dim,
805820
input_sizes.back().data(),
806821
nullptr,
807822
input_dim_order.back().data(),
808-
input_strides.back().data());
823+
input_strides.back().data(),
824+
torch::executor::TensorShapeDynamism::STATIC,
825+
device.type(),
826+
device.index());
809827

810828
torch::executor::Tensor temp =
811829
torch::executor::Tensor(&input_tensors.back());
@@ -1128,18 +1146,34 @@ struct PyMethod final {
11281146
at_tensor.dim() == 4) {
11291147
dim_order = decltype(dim_order)({0, 2, 3, 1});
11301148
} else {
1131-
auto error_msg = "Input " + std::to_string(i) + "for method " +
1149+
auto error_msg = "Input " + std::to_string(i) + " for method " +
11321150
method_->method_meta().name() +
11331151
" should be contiguous or channels-last.";
11341152
throw std::runtime_error(error_msg);
11351153
}
1154+
// Record where the buffer actually lives. The conversion copied every
1155+
// other property and left the device at CPU, so an accelerator buffer
1156+
// was described as host memory.
1157+
auto mapped_device = torch_to_executorch_device(at_tensor.device());
1158+
// An empty tensor has no buffer for a label to describe, so it is left
1159+
// alone rather than rejected for a device it never touches.
1160+
if (!mapped_device.has_value() && at_tensor.numel() != 0) {
1161+
throw std::runtime_error(
1162+
"Input " + std::to_string(i) + " for method " +
1163+
method_->method_meta().name() + " is on device " +
1164+
at_tensor.device().str() +
1165+
", and only CPU and CUDA tensors can be passed to a method.");
1166+
}
1167+
const auto device =
1168+
mapped_device.value_or(aten::Device(aten::DeviceType::CPU));
11361169
TensorPtr tensor = for_blob(
11371170
mutable_tensor_data_ptr_no_cow(at_tensor),
11381171
std::move(sizes),
11391172
type)
11401173
.strides(std::move(strides))
11411174
.dim_order(std::move(dim_order))
11421175
.dynamism(aten::TensorShapeDynamism::STATIC)
1176+
.device(device)
11431177
.make_tensor_ptr();
11441178
input_tensors.push_back(tensor);
11451179
EValue evalue(input_tensors.back());

‎extension/pybindings/test/test_pybindings.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,9 @@ def setUp(self):
5050
print("can't load aten lib")
5151

5252
assert kernel_mode is not None
53+
# Only the portable build converts an incoming tensor, so a test for
54+
# that conversion has to skip under the aten build.
55+
self.kernel_mode = kernel_mode
5356
self.load_fn = runtime._load_for_executorch_from_buffer
5457
self.load_prog_fn = runtime._load_program_from_buffer
5558
self.runtime = runtime
@@ -733,3 +736,53 @@ def test_program_data_separation(self) -> None:
733736
)
734737
with self.assertRaises(RuntimeError):
735738
executorch_module_bundled_no_data.forward(inputs)
739+
740+
def test_method_rejects_input_on_unrepresentable_device(self):
741+
# The conversion used to label every input CPU, so an input whose memory
742+
# the host cannot read was described as host memory, and the failure
743+
# surfaced later as a crash instead of an error naming the input.
744+
if self.kernel_mode != "portable":
745+
self.skipTest("only the portable build converts the input tensor")
746+
747+
exported_program, inputs = create_program(ModuleAdd())
748+
executorch_module = self.load_fn(exported_program.buffer)
749+
750+
with self.assertRaises(RuntimeError) as caught:
751+
executorch_module.forward([inputs[0].to("meta"), inputs[1]])
752+
message = str(caught.exception)
753+
# Asserts the device is named as Python spells it, since an uppercased or
754+
# index-less name would not match what the caller passed.
755+
self.assertIn("is on device meta", message)
756+
self.assertIn("only CPU and CUDA tensors", message)
757+
758+
def test_method_accepts_a_cpu_input_after_the_device_check(self):
759+
# The rejection tests above pass for a change that throws on every input, so this
760+
# asserts the other half: an ordinary CPU input still converts and runs. Without it
761+
# the pair does not distinguish "rejects what it cannot represent" from "rejects
762+
# everything".
763+
exported_program, inputs = create_program(ModuleAdd())
764+
executorch_module = self.load_fn(exported_program.buffer)
765+
766+
executorch_output = executorch_module.forward(inputs)[0]
767+
768+
self.assertTrue(
769+
torch.allclose(executorch_output, inputs[0] + inputs[1]),
770+
"a CPU input must still reach the method and produce the same result",
771+
)
772+
773+
def test_program_method_rejects_input_on_unrepresentable_device(self):
774+
# The other conversion site, reached through a loaded program rather than a
775+
# module. It got the same treatment, and without this it had no test: before
776+
# the change this path aborted the process rather than raising.
777+
if self.kernel_mode != "portable":
778+
self.skipTest("only the portable build converts the input tensor")
779+
780+
exported_program, inputs = create_program(ModuleAdd())
781+
program = self.load_prog_fn(exported_program.buffer)
782+
method = program.load_method("forward")
783+
784+
with self.assertRaises(RuntimeError) as caught:
785+
method.set_inputs([inputs[0].to("meta"), inputs[1]])
786+
message = str(caught.exception)
787+
self.assertIn("is on device meta", message)
788+
self.assertIn("only CPU and CUDA tensors", message)

0 commit comments

Comments
 (0)