Repository navigation
Commit a489948
[ET-VK] Clamp the tanh argument in the gelu shader
The gelu shader evaluates the tanh approximation with an unclamped
argument. For an input of x the argument is
sqrt(2/pi) * (x + 0.044715 * x^3)
which grows cubically, so x = -13.24 already yields -93.4. A driver
that evaluates tanh as (e^y - e^-y) / (e^y + e^-y) overflows fp32 at
|y| > ~88 and returns inf/inf = NaN. On a Mali-G76 this makes the
Whisper encoder emit NaN from conv2 for exactly the two activations
below -13, and the first LayerNorm then propagates them across the
whole tensor.
The tanh op in this same file already clamps to +/-15 for this reason;
gelu now does the same. The clamp is numerically free: 1 - tanh(15) is
1.9e-13, well below fp32 epsilon, so the result is bit-identical for
every input that gets clamped.1 parent f3dd4b4 commit a489948
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| |||
0 commit comments