Skip to content

Commit 2244dc3

Browse files
ssjiaSS-JIA
authored andcommitted
[ET-VK][export] Update tensor representation sync logic to allow for flexibility in memory layouts
Pull Request resolved: pytorch#17564 The tag_memory_meta_pass determines which storage type and memory layout to use for each tensor in the graph. Previously, OpRepSets enforced that "synced" tensors (e.g. all inputs to a binary op) use the exact same storage type AND memory layout by collapsing them into a single shared TensorRepSet. This was overly restrictive for quantized operators like q8ta_add, where inputs and outputs must share the same packed dimension but are allowed to use different memory layouts (e.g. input A uses PACKED_INT8_4W4C, input B uses PACKED_INT8_4C1W, output uses PACKED_INT8_4C1W). This diff introduces PackedDimInfo, a Python-side mirror of the C++ PackedDimInfo struct in Tensor.h, which captures the packed dimension and block size for each memory layout. The sync logic is rewritten so that synced tensors are constrained to have "compatible" packed dim info (same packed_dim and packed_dim_block_size) rather than identical memory layouts. This is achieved through three new TensorRepSet methods: has_same_packed_dim_info_set checks exact PDI equality, has_compatible_packed_dim_info_set checks superset containment, and filter_for_compatible_packed_dim_infos narrows a repset to only layouts with compatible PDIs. The OpRepSets initialization now stores individual repsets per arg/output instead of collapsing synced groups into a single object, and constraint propagation uses packed-dim filtering. The tag_memory_meta_pass is simplified to always call constrain_op_out_repset since the new OpRepSets sync logic handles propagation internally. Also renames make_filtered_tensor_repset to filter_invalid_reprs for clarity and adds comprehensive unit tests for TensorRepSet, TensorRepSetList, OpRepSets, and TensorReprList. Authored with Claude. ghstack-source-id: 343460524 @exported-using-ghexport Differential Revision: [D93768636](https://our.internmc.facebook.com/intern/diff/D93768636/)
1 parent 9a58ce8 commit 2244dc3

7 files changed

Lines changed: 1309 additions & 88 deletions

File tree

‎backends/vulkan/_passes/tag_memory_meta_pass.py‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -394,18 +394,11 @@ def constrain_op_out_repset(self, op_repsets: utils.OpRepSets) -> None:
394394
op_repsets.try_constrain_with_out_repset(out_respset)
395395

396396
def constrain_op_repsets(self, op_repsets: utils.OpRepSets) -> None:
397-
# For most ops, constraining the argument repsets will also contrain the output
398-
# repset due to OpRepSets maintaining synchronization rules.
399397
for i in range(len(op_repsets.op_node.args)):
400398
if utils.is_tensor_arg_node(op_repsets.op_node.args[i]):
401399
self.constrain_op_arg_repset(i, op_repsets)
402400

403-
# However, some operators do not sync input and output representations and also
404-
# define ambiguous repsets for the output tensor(s). In those cases we will need
405-
# to execute additional logic to constrain the output repsets separately from
406-
# the input repsets.
407-
if not op_repsets.sync_primary_io_repr and op_repsets.sync_outs_repr:
408-
self.constrain_op_out_repset(op_repsets)
401+
self.constrain_op_out_repset(op_repsets)
409402

410403
def set_op_node_tensor_reprs(
411404
self, graph_module: torch.fx.GraphModule, op_node: torch.fx.Node

‎backends/vulkan/runtime/VulkanBackend.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,8 @@ utils::GPUMemoryLayout get_memory_layout(
144144
return utils::kPackedInt8_4W4C;
145145
case vkgraph::VkMemoryLayout::PACKED_INT8_4H4W:
146146
return utils::kPackedInt8_4H4W;
147+
case vkgraph::VkMemoryLayout::PACKED_INT8_4W:
148+
return utils::kPackedInt8_4W;
147149
case vkgraph::VkMemoryLayout::PACKED_INT8_4C1W:
148150
return utils::kPackedInt8_4C1W;
149151
default:

‎backends/vulkan/serialization/schema.fbs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ enum VkMemoryLayout : ubyte {
4242
TENSOR_CHANNELS_PACKED = 2,
4343
PACKED_INT8_4W4C = 3,
4444
PACKED_INT8_4H4W = 4,
45+
PACKED_INT8_4W = 5,
4546
PACKED_INT8_4C1W = 8,
4647
DEFAULT_LAYOUT = 255,
4748
}

‎backends/vulkan/serialization/vulkan_graph_schema.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ class VkMemoryLayout(IntEnum):
5050
TENSOR_CHANNELS_PACKED = 2
5151
PACKED_INT8_4W4C = 3
5252
PACKED_INT8_4H4W = 4
53+
PACKED_INT8_4W = 5
5354
PACKED_INT8_4C1W = 8
5455
DEFAULT_LAYOUT = 255
5556

‎backends/vulkan/test/TARGETS‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,17 @@ python_unittest(
6060
],
6161
)
6262

63+
python_unittest(
64+
name = "test_vulkan_tensor_repr",
65+
srcs = [
66+
"test_vulkan_tensor_repr.py",
67+
],
68+
deps = [
69+
"//caffe2:torch",
70+
"//executorch/backends/vulkan:vulkan_preprocess",
71+
],
72+
)
73+
6374
runtime.python_library(
6475
name = "tester",
6576
srcs = ["tester.py"],

0 commit comments

Comments
 (0)