Repository navigation
Commit 2f964f8
authored
[ET-VK] Scale softmax workers with the row length and reduce as a tree (pytorch#22349)
`softmax_buffer.glsl` hardcodes
```glsl
#define NWORKERS 4
#define MAX_NTHREADS 16
```
and `pick_softmax_gwg()` matches it with `lwg_extents[reduce_dim] = 4u`,
so four threads normalise a row no matter how long it is. On
`all-MiniLM-L6-v2` at its published 254-token shape that is six
dispatches of 633 us each: **21.2% of GPU time** on an Adreno 840.
Widening alone only recovered a quarter of it. The aggregation after
each barrier was a serial walk of all `NWORKERS` shared entries executed
by **every thread**, not just thread 0, so its cost grew with the worker
count and cancelled most of the benefit. Replacing both walks (max, then
sum) with a shared-memory tree is the larger half of the win. The loop
bound is uniform and `barrier()` sits outside the guard, so every thread
still reaches every barrier.
### Measurements
`all-MiniLM-L6-v2` @254 on an Adreno 840, four interleaved
order-reversed rounds, best of 20 executions each:
| | best |
| --- | --- |
| baseline | 18.30 ms |
| + workers scaled | 17.40 ms |
| + tree aggregation | **15.53 ms** (-15.1%) |
`softmax_buffer` itself goes 3.80 -> 2.91 ms. Output matches the XNNPACK
build to cosine 0.9999975 and is 60/60 bit-identical across runs.
### Notes
`reduce_dim` is a WHCN/xyz index (0 = x = last dim) while `size_at()`
counts back from the end, hence the `-(reduce_dim + 1)` in
`softmax_nworkers()`.
Buffer storage only. The texture path uses a different shader and
grouping scheme and is left alone; it has the same hardcoded constants
and is worth a follow-up.
Fixes pytorch#22351
cc @SS-JIA @manuelcandales @digantdesai @cbilgin1 parent 5fc01ad commit 2f964f8
2 files changed
Lines changed: 95 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
38 | 41 | | |
39 | | - | |
40 | | - | |
| 42 | + | |
| 43 | + | |
41 | 44 | | |
42 | 45 | | |
43 | 46 | | |
| |||
81 | 84 | | |
82 | 85 | | |
83 | 86 | | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
88 | 97 | | |
| 98 | + | |
89 | 99 | | |
90 | 100 | | |
91 | 101 | | |
| |||
97 | 107 | | |
98 | 108 | | |
99 | 109 | | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
104 | 116 | | |
105 | 117 | | |
106 | | - | |
| 118 | + | |
107 | 119 | | |
108 | 120 | | |
109 | 121 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
21 | 67 | | |
22 | 68 | | |
23 | 69 | | |
| |||
36 | 82 | | |
37 | 83 | | |
38 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
39 | 92 | | |
40 | 93 | | |
41 | 94 | | |
| |||
95 | 148 | | |
96 | 149 | | |
97 | 150 | | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | 151 | | |
105 | 152 | | |
106 | | - | |
107 | | - | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
108 | 170 | | |
109 | | - | |
110 | 171 | | |
111 | 172 | | |
112 | 173 | | |
| |||
0 commit comments