Skip to content

Commit 55d693b

Browse files
authored
tensor_parser_aten: tag planned tensor DataPtr with its real device (pytorch#21289)
Summary: In ATen mode, parseTensor built every planned tensor's storage DataPtr with a hardcoded c10::DeviceType::CPU, and its options with at::CPU(type). A tensor whose planned buffer lives on an accelerator (e.g. a CUDA-delegate planned buffer) therefore got a CPU-tagged data pointer, so the runtime treated device memory as host memory and the delegate rejected it (Method::init fails in getDeviceFromPtr on a CPU-tagged device pointer). Fix: read the device from the serialized tensor's extra_tensor_info (device_type/device_index, defaulting to CPU when absent, matching the portable parser), then build the tensor with a single at::from_blob(ptr, sizes, strides, /*storage_offset=*/0, deleteNothing, at::TensorOptions().dtype(type).device(device), /*target_device=*/device) so the storage DataPtr, the TensorImpl device (device(), is_cuda()), and the dispatch key all agree. Passing target_device makes from_blob skip getDeviceFromPtr, so the same path works for a real device pointer and for a null runtime-bound pointer without inspecting the pointer. TRT-only / CPU programs are unchanged (device defaults to CPU). The serialized device_type is mapped to its ATen counterpart through a small table so validation and conversion share one source of truth and a new device only needs one entry. The untrusted device_type and device_index are validated (unmapped type or a negative accelerator index is rejected), and a non-CPU DYNAMIC_UNBOUND tensor is rejected rather than silently returning CPU-resizable storage that contradicts the serialized device. Also update internal_set_tensor_data (tensor_util_aten.cpp) to preserve the source tensor's device instead of hardcoding CPU, and verify the destination and source devices match before sharing storage, so a shared tensor cannot end up with a TensorImpl device that disagrees with its DataPtr device. fbcode and xplat copies kept identical. Test Plan: New ATen-mode unit test for the device parsing logic: buck2 test fbcode//executorch/runtime/executor/test:tensor_parser_aten_test Pass 6, fail 0. Covers CPU tensors staying unindexed, missing extra_tensor_info defaulting to CPU, rejection of an unmapped device_type, rejection of a negative accelerator index, rejection of DYNAMIC_UNBOUND on a non-CPU device, and DYNAMIC_UNBOUND still allowed on CPU. ATen-mode load + execute path (the CPU device-index regression this restores): buck2 test fbcode//executorch/extension/module/test:module_test_aten Pass 50, fail 0. This exercises parseTensor followed by aten kernels (e.g. aten::mul.out) on ModuleAddMul; a hardcoded CPU index 0 previously made ATen see cpu vs cpu:0 and abort the kernel, which this fix avoids by keeping CPU unindexed. Unblocks loading a coalesced TRT+CUDA .pte whose CUDA-delegate planned buffers are device memory.
1 parent 16b780b commit 55d693b

4 files changed

Lines changed: 282 additions & 7 deletions

File tree

‎runtime/core/exec_aten/util/tensor_util_aten.cpp‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -135,9 +135,17 @@ Error share_tensor_data(const at::Tensor& t_dst, const at::Tensor& t_src) {
135135
t_src.mutable_data_ptr() != nullptr,
136136
InvalidArgument,
137137
"Source tensor should have data_ptr not being nullptr.");
138-
// Assign the dataptr as the input tensor dataptr
139-
storage->set_data_ptr(
140-
at::DataPtr(t_src.mutable_data_ptr(), at::DeviceType::CPU));
138+
// The destination keeps its own TensorImpl but adopts the source storage, so
139+
// the devices must agree or TensorImpl and DataPtr would disagree.
140+
ET_CHECK_OR_RETURN_ERROR(
141+
t_dst.device() == t_src.device(),
142+
InvalidArgument,
143+
"Destination device %s does not match source device %s",
144+
c10::str(t_dst.device()).c_str(),
145+
c10::str(t_src.device()).c_str());
146+
// Preserve the source device; hardcoding CPU would mis-tag a device input's
147+
// storage as host and the backend would later reject it.
148+
storage->set_data_ptr(at::DataPtr(t_src.mutable_data_ptr(), t_src.device()));
141149
storage->set_nbytes(t_src.nbytes());
142150

143151
return Error::Ok;

‎runtime/executor/tensor_parser_aten.cpp‎

Lines changed: 72 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,19 @@ namespace {
3030
void deleteNothing(void*);
3131
void deleteNothing(void*) {}
3232

33+
// Maps a serialized DeviceType to its ATen counterpart. Kept as a table so
34+
// validation and conversion share one source of truth and a new device only
35+
// needs a single entry here.
36+
struct DeviceTypeMapping {
37+
executorch_flatbuffer::DeviceType serialized;
38+
c10::DeviceType aten;
39+
};
40+
41+
constexpr DeviceTypeMapping kDeviceTypeMappings[] = {
42+
{executorch_flatbuffer::DeviceType::CPU, c10::DeviceType::CPU},
43+
{executorch_flatbuffer::DeviceType::CUDA, c10::DeviceType::CUDA},
44+
};
45+
3346
} // namespace
3447

3548
Result<at::Tensor> parseTensor(
@@ -53,6 +66,47 @@ Result<at::Tensor> parseTensor(
5366
InvalidProgram,
5467
"Invalid ScalarType %" PRId8,
5568
static_cast<int8_t>(type));
69+
70+
// Defaults to CPU when extra_tensor_info is absent (older PTE files). A
71+
// device-delegate planned buffer must be tagged with its real device or the
72+
// runtime treats device memory as host memory.
73+
c10::DeviceType device_type = c10::DeviceType::CPU;
74+
c10::DeviceIndex device_index = 0;
75+
if (s_tensor->extra_tensor_info() != nullptr) {
76+
// Untrusted byte from the PTE; an unmapped value is rejected below so it
77+
// cannot reach c10::Device as garbage.
78+
const auto raw_device_type = s_tensor->extra_tensor_info()->device_type();
79+
bool valid_device_type = false;
80+
for (const auto& mapping : kDeviceTypeMappings) {
81+
if (mapping.serialized == raw_device_type) {
82+
device_type = mapping.aten;
83+
valid_device_type = true;
84+
break;
85+
}
86+
}
87+
ET_CHECK_OR_RETURN_ERROR(
88+
valid_device_type,
89+
InvalidProgram,
90+
"Invalid DeviceType %" PRId8,
91+
static_cast<int8_t>(raw_device_type));
92+
device_index = static_cast<c10::DeviceIndex>(
93+
s_tensor->extra_tensor_info()->device_index());
94+
// Reject a negative accelerator index from the untrusted PTE; -1
95+
// (any/current device) is not a valid serialized placement and would later
96+
// confuse device matching.
97+
ET_CHECK_OR_RETURN_ERROR(
98+
device_type == c10::DeviceType::CPU || device_index >= 0,
99+
InvalidProgram,
100+
"Invalid device_index %" PRId8,
101+
static_cast<int8_t>(device_index));
102+
}
103+
// CPU stays unindexed: an explicit cpu:0 would mismatch the graph's default
104+
// cpu tensors and trip ATen's same-device check. Only accelerators carry an
105+
// index.
106+
const c10::Device device = device_type == c10::DeviceType::CPU
107+
? c10::Device(device_type)
108+
: c10::Device(device_type, device_index);
109+
// Sized with null data to compute nbytes; real device is applied below.
56110
auto options = at::CPU(type).options();
57111

58112
ET_CHECK_OR_RETURN_ERROR(
@@ -103,8 +157,13 @@ Result<at::Tensor> parseTensor(
103157

104158
if (s_tensor->shape_dynamism() ==
105159
executorch_flatbuffer::TensorShapeDynamism::DYNAMIC_UNBOUND) {
106-
// Provide fully dynamic tensors with an allocator so they can be resized
107-
// within aten kernels.
160+
// Fully dynamic tensors get a CPU allocator so aten kernels can resize
161+
// them. A device tensor cannot be honored here, so reject it rather than
162+
// silently returning CPU storage that contradicts the serialized device.
163+
ET_CHECK_OR_RETURN_ERROR(
164+
device_type == c10::DeviceType::CPU,
165+
NotSupported,
166+
"DYNAMIC_UNBOUND tensors are only supported on CPU");
108167
auto impl = tensor.unsafeGetTensorImpl();
109168
at::StorageImpl* storage = impl->unsafe_storage().unsafeGetStorageImpl();
110169
storage->set_allocator(at::getCPUAllocator());
@@ -128,8 +187,17 @@ Result<at::Tensor> parseTensor(
128187
static_cast<uint32_t>(data_ptr.error()));
129188
return data_ptr.error();
130189
}
131-
tensor.unsafeGetTensorImpl()->unsafe_storage().set_data_ptr(
132-
at::DataPtr(data_ptr.get(), c10::DeviceType::CPU));
190+
// Rebuild so storage DataPtr, TensorImpl device, and dispatch key agree.
191+
// target_device makes from_blob skip getDeviceFromPtr, so the same path
192+
// works for a real pointer and for a null runtime-bound one.
193+
tensor = at::from_blob(
194+
data_ptr.get(),
195+
sizes,
196+
strides,
197+
/*storage_offset=*/0,
198+
deleteNothing,
199+
at::TensorOptions().dtype(type).device(device),
200+
/*target_device=*/device);
133201
}
134202

135203
return tensor;

‎runtime/executor/test/targets.bzl‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,19 @@ def define_common_targets(is_fbcode = False):
319319
env = modules_env,
320320
)
321321

322+
runtime.cxx_test(
323+
name = "tensor_parser_aten_test",
324+
srcs = [
325+
"tensor_parser_aten_test.cpp",
326+
],
327+
deps = [
328+
":managed_memory_manager",
329+
"//executorch/runtime/executor:program_aten",
330+
"//executorch/runtime/core/exec_aten:lib_aten",
331+
"//executorch/schema:program",
332+
],
333+
)
334+
322335
runtime.cxx_test(
323336
name = "tensor_parser_device_test",
324337
srcs = [
Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
/**
10+
* Unit tests for the ATen-mode tensor parser's device handling. These exercise
11+
* the validation and CPU-tagging paths that run without a real accelerator. The
12+
* end-to-end CUDA device-pointer path is covered separately by
13+
* tensor_parser_device_test (portable mode) with a mock allocator; ATen-mode
14+
* CUDA execution needs a real device and is out of scope here.
15+
*/
16+
17+
#include <executorch/runtime/executor/tensor_parser.h>
18+
19+
#include <executorch/runtime/core/exec_aten/exec_aten.h>
20+
#include <executorch/runtime/executor/test/managed_memory_manager.h>
21+
#include <executorch/runtime/platform/runtime.h>
22+
#include <executorch/schema/program_generated.h>
23+
24+
#include <gtest/gtest.h>
25+
26+
#include <vector>
27+
28+
using executorch::aten::Tensor;
29+
using executorch::runtime::Error;
30+
using executorch::runtime::MemoryManager;
31+
using executorch::runtime::Result;
32+
using executorch::runtime::aten::deserialization::parseTensor;
33+
using executorch::runtime::testing::ManagedMemoryManager;
34+
35+
namespace {
36+
37+
constexpr size_t kMemBytes = 32 * 1024U;
38+
39+
// Builds a standalone serialized Tensor with no data_buffer and no
40+
// allocation_info, i.e. an input/placeholder. parseTensor then resolves a null
41+
// data pointer, so these tests reach the device validation and tagging logic
42+
// without needing a Program or any device memory.
43+
const executorch_flatbuffer::Tensor* buildTensor(
44+
flatbuffers::FlatBufferBuilder& builder,
45+
executorch_flatbuffer::TensorShapeDynamism dynamism,
46+
bool with_extra_info,
47+
executorch_flatbuffer::DeviceType device_type,
48+
int8_t device_index) {
49+
flatbuffers::Offset<executorch_flatbuffer::ExtraTensorInfo> extra_info = 0;
50+
if (with_extra_info) {
51+
executorch_flatbuffer::ExtraTensorInfoBuilder eti(builder);
52+
eti.add_device_type(device_type);
53+
eti.add_device_index(device_index);
54+
extra_info = eti.Finish();
55+
}
56+
const std::vector<int32_t> sizes = {2, 2};
57+
const std::vector<uint8_t> dim_order = {0, 1};
58+
const auto sizes_offset = builder.CreateVector(sizes);
59+
const auto dim_order_offset = builder.CreateVector(dim_order);
60+
61+
executorch_flatbuffer::TensorBuilder tb(builder);
62+
tb.add_scalar_type(executorch_flatbuffer::ScalarType::FLOAT);
63+
tb.add_storage_offset(0);
64+
tb.add_sizes(sizes_offset);
65+
tb.add_dim_order(dim_order_offset);
66+
tb.add_data_buffer_idx(0);
67+
tb.add_shape_dynamism(dynamism);
68+
if (with_extra_info) {
69+
tb.add_extra_tensor_info(extra_info);
70+
}
71+
builder.Finish(tb.Finish());
72+
return flatbuffers::GetRoot<executorch_flatbuffer::Tensor>(
73+
builder.GetBufferPointer());
74+
}
75+
76+
} // namespace
77+
78+
class TensorParserAtenTest : public ::testing::Test {
79+
protected:
80+
void SetUp() override {
81+
executorch::runtime::runtime_init();
82+
}
83+
};
84+
85+
// A CPU tensor must stay unindexed. An explicit cpu:0 would mismatch the
86+
// graph's default cpu tensors and trip ATen's same-device check.
87+
TEST_F(TensorParserAtenTest, CpuTensorIsUnindexed) {
88+
flatbuffers::FlatBufferBuilder builder;
89+
const auto* s_tensor = buildTensor(
90+
builder,
91+
executorch_flatbuffer::TensorShapeDynamism::STATIC,
92+
/*with_extra_info=*/true,
93+
executorch_flatbuffer::DeviceType::CPU,
94+
/*device_index=*/0);
95+
96+
ManagedMemoryManager mmm(kMemBytes, kMemBytes);
97+
Result<Tensor> tensor = parseTensor(nullptr, &mmm.get(), s_tensor);
98+
ASSERT_EQ(tensor.error(), Error::Ok);
99+
EXPECT_TRUE(tensor->is_cpu());
100+
EXPECT_FALSE(tensor->device().has_index());
101+
EXPECT_EQ(tensor->scalar_type(), executorch::aten::ScalarType::Float);
102+
}
103+
104+
// A tensor with no extra_tensor_info (older PTE files) defaults to CPU.
105+
TEST_F(TensorParserAtenTest, MissingExtraInfoDefaultsToCpu) {
106+
flatbuffers::FlatBufferBuilder builder;
107+
const auto* s_tensor = buildTensor(
108+
builder,
109+
executorch_flatbuffer::TensorShapeDynamism::STATIC,
110+
/*with_extra_info=*/false,
111+
executorch_flatbuffer::DeviceType::CPU,
112+
/*device_index=*/0);
113+
114+
ManagedMemoryManager mmm(kMemBytes, kMemBytes);
115+
Result<Tensor> tensor = parseTensor(nullptr, &mmm.get(), s_tensor);
116+
ASSERT_EQ(tensor.error(), Error::Ok);
117+
EXPECT_TRUE(tensor->is_cpu());
118+
EXPECT_FALSE(tensor->device().has_index());
119+
}
120+
121+
// An unmapped DeviceType from the untrusted PTE is rejected before it can reach
122+
// c10::Device as garbage.
123+
TEST_F(TensorParserAtenTest, InvalidDeviceTypeIsRejected) {
124+
flatbuffers::FlatBufferBuilder builder;
125+
const auto* s_tensor = buildTensor(
126+
builder,
127+
executorch_flatbuffer::TensorShapeDynamism::STATIC,
128+
/*with_extra_info=*/true,
129+
static_cast<executorch_flatbuffer::DeviceType>(7),
130+
/*device_index=*/0);
131+
132+
ManagedMemoryManager mmm(kMemBytes, kMemBytes);
133+
EXPECT_EQ(
134+
parseTensor(nullptr, &mmm.get(), s_tensor).error(),
135+
Error::InvalidProgram);
136+
}
137+
138+
// A negative accelerator index is not a valid serialized placement.
139+
TEST_F(TensorParserAtenTest, NegativeAcceleratorIndexIsRejected) {
140+
flatbuffers::FlatBufferBuilder builder;
141+
const auto* s_tensor = buildTensor(
142+
builder,
143+
executorch_flatbuffer::TensorShapeDynamism::STATIC,
144+
/*with_extra_info=*/true,
145+
executorch_flatbuffer::DeviceType::CUDA,
146+
/*device_index=*/-1);
147+
148+
ManagedMemoryManager mmm(kMemBytes, kMemBytes);
149+
EXPECT_EQ(
150+
parseTensor(nullptr, &mmm.get(), s_tensor).error(),
151+
Error::InvalidProgram);
152+
}
153+
154+
// A fully dynamic tensor gets a CPU-resizable allocator, so a non-CPU device
155+
// cannot be honored and must be rejected rather than silently tagged CPU.
156+
TEST_F(TensorParserAtenTest, DynamicUnboundCudaIsRejected) {
157+
flatbuffers::FlatBufferBuilder builder;
158+
const auto* s_tensor = buildTensor(
159+
builder,
160+
executorch_flatbuffer::TensorShapeDynamism::DYNAMIC_UNBOUND,
161+
/*with_extra_info=*/true,
162+
executorch_flatbuffer::DeviceType::CUDA,
163+
/*device_index=*/0);
164+
165+
ManagedMemoryManager mmm(kMemBytes, kMemBytes);
166+
EXPECT_EQ(
167+
parseTensor(nullptr, &mmm.get(), s_tensor).error(), Error::NotSupported);
168+
}
169+
170+
// The DYNAMIC_UNBOUND guard only rejects non-CPU; CPU resizable tensors still
171+
// parse so the guard does not regress the common case.
172+
TEST_F(TensorParserAtenTest, DynamicUnboundCpuIsAllowed) {
173+
flatbuffers::FlatBufferBuilder builder;
174+
const auto* s_tensor = buildTensor(
175+
builder,
176+
executorch_flatbuffer::TensorShapeDynamism::DYNAMIC_UNBOUND,
177+
/*with_extra_info=*/false,
178+
executorch_flatbuffer::DeviceType::CPU,
179+
/*device_index=*/0);
180+
181+
ManagedMemoryManager mmm(kMemBytes, kMemBytes);
182+
Result<Tensor> tensor = parseTensor(nullptr, &mmm.get(), s_tensor);
183+
ASSERT_EQ(tensor.error(), Error::Ok);
184+
EXPECT_TRUE(tensor->is_cpu());
185+
EXPECT_FALSE(tensor->device().has_index());
186+
}

0 commit comments

Comments
 (0)