Skip to content

Commit d62d70f

Browse files
authored
Vulkan: stop delegating nodes past their own support check (pytorch#21847)
### Summary Two Vulkan backend fixes found while bringing up LFM2.5-VL (a hybrid attention + gated short-conv VLM) on Android. Neither is specific to that model. **1. Nodes were delegated past their own support check.** Concretely: a rotary embedding's `freqs_cos[input_pos]` feeds the `hf_rope` pattern, so its `aten.index.Tensor` was delegated even though `check_index_tensor_node` rejects a non-1-D `self`. The gather then read `self[idx, 0, 0, 0]` against a `(4096, 64)` table, and `resize_index_tensor_node` set rank-1 sizes on a rank-2 tensor, aborting in `virtual_resize` with *"new sizes cannot modify the dimensionality of the tensor"*. This PR excludes the placeholder bindings from the match. The texture shader still assumes the 1-D form, so `pick_io_storage_fn` routes a higher-rank `self` to a contiguous buffer. **2. `VulkanBackend::execute` underflows on a delegate arg-count mismatch.** `output_offset = args.size() - num_outputs` is `size_t`. If the serialized graph declares more outputs than the delegate call has arguments, that subtraction wraps and every output access reads through a wild pointer — the failure surfaces as a SIGSEGV inside `execute()` with nothing pointing at the cause. The reachable route today is a mutated buffer serialized as a graph output. LFM2.5's ten short-conv states declared 11 outputs against a 3-argument call. This PR checks the counts up front so the result is a readable error naming both numbers rather than a segfault. It does not change the mutation handling itself. ### Test plan - New `test_vulkan_backend_index_tensor_higher_rank_self` in `backends/vulkan/test/test_vulkan_delegate.py`, covering `table[positions]` with a 2-D table — the shape a rotary embedding uses. This fails before the first change and passes after. - Existing `backends/vulkan/test/test_vulkan_delegate.py` suite. - On device (Galaxy S26 Ultra, Adreno, driver 0842.19.8): LFM2.5-VL 450M and 1.6B keep the rotary embedding inside the delegate, lower to a single Vulkan call for the decoder, and generate correct text and image responses. Before the first change the export aborted in `virtual_resize`; with it, decode runs at ~150 tok/s (450M) and ~72 tok/s (1.6B) versus ~100 and ~54 on XNNPACK, and text-only TTFT drops from 326→~185 ms and 565→~304 ms respectively. cc @SS-JIA @manuelcandales @digantdesai @cbilgin Fixes pytorch#21876
1 parent be8df33 commit d62d70f

6 files changed

Lines changed: 112 additions & 18 deletions

File tree

‎backends/vulkan/op_registry.py‎

Lines changed: 33 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1437,33 +1437,56 @@ def register_where():
14371437

14381438
@update_features(exir_ops.edge.aten.index.Tensor)
14391439
def register_index_tensor():
1440-
def check_index_tensor_node(node: torch.fx.Node) -> bool:
1440+
def _index_tensor_shapes(node: torch.fx.Node):
1441+
"""(self_val, index_val) for the supported single-index form, else None."""
14411442
self_arg = node.args[0]
14421443
indices = node.args[1]
14431444

1444-
# Only support 1D self tensor
14451445
if not isinstance(self_arg, torch.fx.Node):
1446-
return False
1446+
return None
14471447
self_val = self_arg.meta.get("val", None)
14481448
if self_val is None:
1449-
return False
1450-
if len(self_val.size()) != 1:
1451-
return False
1449+
return None
14521450

1453-
# Only support exactly one non-None index tensor
1451+
# Only support exactly one non-None index tensor, applied to dim 0.
14541452
if not isinstance(indices, (list, tuple)):
1455-
return False
1453+
return None
14561454
non_none = [idx for idx in indices if idx is not None]
1457-
if len(non_none) != 1:
1455+
if len(non_none) != 1 or indices[0] is None:
1456+
return None
1457+
index_arg = non_none[0]
1458+
if not isinstance(index_arg, torch.fx.Node):
1459+
return None
1460+
index_val = index_arg.meta.get("val", None)
1461+
if index_val is None:
1462+
return None
1463+
1464+
return self_val, index_val
1465+
1466+
def check_index_tensor_node(node: torch.fx.Node) -> bool:
1467+
shapes = _index_tensor_shapes(node)
1468+
if shapes is None:
14581469
return False
1470+
_, index_val = shapes
1471+
# The gather is expressed as "one index position per output slice", so
1472+
# the index must be 1-D. `self` may be any rank: the buffer shader
1473+
# copies self's trailing dims through unchanged.
1474+
return len(index_val.size()) == 1
14591475

1460-
return True
1476+
def pick_index_tensor_storage(node: torch.fx.Node):
1477+
shapes = _index_tensor_shapes(node)
1478+
# Only the buffer shader handles a higher-rank `self`; the texture
1479+
# variant still assumes the 1-D form (it reads self[idx, 0, 0, 0]).
1480+
if shapes is not None and len(shapes[0].size()) > 1:
1481+
return utils.CONTIGUOUS_BUFFER, utils.CONTIGUOUS_BUFFER
1482+
return utils.ANY_STORAGE, utils.ANY_STORAGE
14611483

14621484
return OpFeatures(
14631485
inputs_storage=utils.ANY_STORAGE,
14641486
inputs_dtypes=utils.FP_INT_T,
14651487
supports_resize=True,
14661488
are_node_inputs_supported_fn=check_index_tensor_node,
1489+
pick_io_storage_fn=pick_index_tensor_storage,
14671490
)
14681491

14691492

‎backends/vulkan/patterns/pattern_registry.py‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,29 @@ def __init__(
4242
def create_pattern_match_from_internal_match(
4343
internal_match: InternalMatch,
4444
) -> PatternMatch:
45+
# `nodes_map` maps every pattern node to its match in the target graph,
46+
# INCLUDING the pattern's placeholders. A placeholder's match is the node
47+
# that merely *feeds* the matched subgraph, which the fused op does not
48+
# compute and the partitioner therefore has no reason to claim.
49+
#
50+
# This matters because `all_nodes` becomes VulkanPartitioner's
51+
# `fusable_nodes`, and `_is_node_supported` returns True for any node in
52+
# that set *before* it consults the op's `are_node_inputs_supported_fn`.
53+
# Including the feeder nodes therefore silently waives their support
54+
# checks: an `aten.index.Tensor` feeding the `hf_rope` pattern got
55+
# delegated despite `check_index_tensor_node` rejecting non-1-D `self`,
56+
# and Vulkan's 1-D-only gather then aborted in `virtual_resize` with
57+
# "new sizes cannot modify the dimensionality of the tensor".
58+
placeholder_matches = set(internal_match.placeholder_nodes)
59+
matched_nodes = [
60+
node
61+
for node in internal_match.nodes_map.values()
62+
if node not in placeholder_matches
63+
]
4564
return PatternMatch(
4665
internal_match.placeholder_nodes,
4766
internal_match.returning_nodes,
48-
list(internal_match.nodes_map.values()),
67+
matched_nodes,
4968
)
5069

5170

‎backends/vulkan/runtime/VulkanBackend.cpp‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -719,6 +719,22 @@ class VulkanBackend final : public ::executorch::runtime::BackendInterface {
719719

720720
const size_t num_inputs = compute_graph->inputs().size();
721721
const size_t num_outputs = compute_graph->outputs().size();
722+
// `args` carries the delegate call's inputs followed by its outputs. If the
723+
// serialized graph disagrees about either count, the `args.size() -
724+
// num_outputs` computed below underflows and every output access reads
725+
// through a wild pointer, so report the mismatch rather than segfault on
726+
// it. A mutated buffer serialized as a graph output is one way to reach
727+
// this: nothing passes such a buffer to the delegate call, so it has no
728+
// argument slot.
729+
VK_CHECK_COND(
730+
args.size() == num_inputs + num_outputs,
731+
"Vulkan graph declares ",
732+
num_inputs,
733+
" inputs and ",
734+
num_outputs,
735+
" outputs, but the delegate call supplied ",
736+
args.size(),
737+
" arguments");
722738
bool should_propagate_resize = false;
723739
#ifdef ET_EVENT_TRACER_ENABLED
724740
runtime::EventTracer* event_tracer = context.event_tracer();

‎backends/vulkan/runtime/graph/ops/glsl/index_tensor_buffer.glsl‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,15 +43,24 @@ void main() {
4343
// Convert output buffer index to tensor index
4444
TensorIndex out_tidx = linear_idx_to_tensor_idx(outp, out_bufi);
4545

46-
// Read the index value at the same tensor position
47-
const uint index_bufi = tensor_idx_to_linear_idx(index, out_tidx);
46+
// aten.index.Tensor with a single index tensor gathers along dim 0. This
47+
// index space is WHCN-ordered -- axis 0 is the LAST pytorch dim -- so
48+
// pytorch dim 0 is the highest axis, and self's trailing dims map 1:1 onto
49+
// out's. With a 1-D index, rank(out) == rank(self).
50+
const uint gather_axis = ndim(outp) - 1;
51+
const uint gather_pos = idx_at(out_tidx, gather_axis);
52+
53+
// The index tensor is 1-D, so only its axis 0 (W) is populated. Indexing it
54+
// with out's full coordinate is equivalent only when out is itself 1-D.
55+
TensorIndex index_tidx;
56+
initialize(index_tidx);
57+
index_tidx.data[0][0] = gather_pos;
58+
const uint index_bufi = tensor_idx_to_linear_idx(index, index_tidx);
4859
const int idx = t_index[index_bufi];
4960

50-
// Construct a tensor index for the 1D self tensor.
51-
// In WHCN ordering, a 1D tensor has its elements along dim 0 (width).
52-
TensorIndex self_tidx;
53-
self_tidx.data[0] = uvec4(uint(idx), 0, 0, 0);
54-
self_tidx.data[1] = uvec4(0);
61+
// self shares out's trailing coordinates; only the gathered axis differs.
62+
TensorIndex self_tidx = out_tidx;
63+
self_tidx.data[div_4(gather_axis)][mod_4(gather_axis)] = uint(idx);
5564
const uint self_bufi = tensor_idx_to_linear_idx(inp, self_tidx);
5665

5766
t_out[out_bufi] = t_self[self_bufi];

‎backends/vulkan/runtime/graph/ops/impl/IndexTensor.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,18 @@ void resize_index_tensor_node(
2020
const std::vector<ValueRef>& resize_args) {
2121
(void)resize_args;
2222
const ValueRef out = args.at(0).refs.at(0);
23+
const ValueRef self = args.at(1).refs.at(0);
2324
const ValueRef index = args.at(1).refs.at(1);
2425

26+
// aten.index.Tensor with a single index tensor gathers along dim 0, so
27+
// out.sizes = index.sizes ++ self.sizes[1:]
28+
// Using the index's sizes alone is only correct when self is 1-D; for any
29+
// higher-rank self it also changes the tensor's RANK, which virtual_resize
30+
// rejects outright ("new sizes cannot modify the dimensionality").
31+
const std::vector<int64_t> self_sizes = graph->sizes_of(self);
2532
std::vector<int64_t> out_sizes = graph->sizes_of(index);
33+
out_sizes.insert(out_sizes.end(), self_sizes.begin() + 1, self_sizes.end());
34+
2635
graph->virtual_resize(out, out_sizes);
2736
}
2837

‎backends/vulkan/test/test_vulkan_delegate.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -762,6 +762,24 @@ def forward(self, x):
762762

763763
self.lower_module_and_test_output(model, sample_inputs)
764764

765+
def test_vulkan_backend_index_tensor_higher_rank_self(self):
766+
# `table[positions]` with a 2-D table, the shape RoPE uses to look up
767+
# its precomputed frequencies. The gather only ever needed `indices` to
768+
# be 1-D -- `self`'s trailing dims come through unchanged -- but the
769+
# support check used to require a 1-D `self` too, and the buffer shader
770+
# only read `self[idx, 0, 0, 0]`.
771+
class Gather(torch.nn.Module):
772+
def __init__(self):
773+
super().__init__()
774+
self.register_buffer("table", torch.rand(32, 8))
775+
776+
def forward(self, positions):
777+
return self.table[positions] * 2.0
778+
779+
sample_inputs = (torch.arange(5, dtype=torch.int32),)
780+
781+
self.lower_module_and_test_output(Gather(), sample_inputs)
782+
765783
@disable_test(
766784
"Currently this test is failing due to weird partitioning because the eq scalar"
767785
"operator is not supported yet. Re-enable when the operator is supported."

0 commit comments

Comments
 (0)