Skip to content

Commit df3d715

Browse files
Qualcomm AI Engine Direct - Change transpose output from NCH1 to NC1W before conv (pytorch#21707)
### Summary This memory layout has better performance on the HTP backend, and lead to better performance on prefill model. #### Prefill performance of third partitioned graph: | Model | w/o | w/ | |---------------------|:--------:|:--------:| | llama3.2 3b instruct| 55924 | 35774 | cc @cbilgin @psiddh
1 parent 43f89fb commit df3d715

1 file changed

Lines changed: 13 additions & 4 deletions

File tree

‎backends/qualcomm/utils/utils.py‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -168,11 +168,20 @@ def __init__(self, weight, bias=None):
168168

169169
def forward(self, x):
170170
rank = x.dim()
171-
x = x.reshape(*x.shape, 1) if rank == 3 else x.reshape(1, *x.shape, 1)
172-
x = torch.transpose(x, 1, 2)
171+
if rank == 3:
172+
bsz, dim0, dim1 = x.size()
173+
x = torch.reshape(x, (bsz, dim0, 1, dim1))
174+
else:
175+
dim0, dim1 = x.size()
176+
x = torch.reshape(x, (1, dim0, 1, dim1))
177+
x = torch.transpose(x, 1, 3)
173178
res = self.conv(x)
174-
res = torch.transpose(res, 1, 2)
175-
res = res.squeeze(-1) if rank == 3 else res.reshape(*res.shape[1:3])
179+
res = res.permute(0, 3, 1, 2)
180+
res = (
181+
res.squeeze(-1)
182+
if rank == 3
183+
else res.reshape(dim0, self.conv.weight.shape[0])
184+
)
176185
return res
177186

178187
def replace_linear(module: torch.nn.Module):

0 commit comments

Comments
 (0)