Skip to content

Commit a489948

Browse files
msluszniakSS-JIA
authored andcommitted
[ET-VK] Clamp the tanh argument in the gelu shader
The gelu shader evaluates the tanh approximation with an unclamped argument. For an input of x the argument is sqrt(2/pi) * (x + 0.044715 * x^3) which grows cubically, so x = -13.24 already yields -93.4. A driver that evaluates tanh as (e^y - e^-y) / (e^y + e^-y) overflows fp32 at |y| > ~88 and returns inf/inf = NaN. On a Mali-G76 this makes the Whisper encoder emit NaN from conv2 for exactly the two activations below -13, and the first LayerNorm then propagates them across the whole tensor. The tanh op in this same file already clamps to +/-15 for this reason; gelu now does the same. The clamp is numerically free: 1 - tanh(15) is 1.9e-13, well below fp32 epsilon, so the result is bit-identical for every input that gets clamped.
1 parent f3dd4b4 commit a489948

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎backends/vulkan/runtime/graph/ops/glsl/unary_op.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ unary_op:
3131
- NAME: exp
3232
OPERATOR: exp(X)
3333
- NAME: gelu
34-
OPERATOR: 0.5 * X * (1 + tanh(sqrt(2 / 3.141593) * (X + 0.044715 * X * X * X)))
34+
OPERATOR: 0.5 * X * (1 + tanh(clamp(sqrt(2 / 3.141593) * (X + 0.044715 * X * X * X), -15.0, 15.0)))
3535
- NAME: neg
3636
OPERATOR: -X
3737
- NAME: sigmoid

0 commit comments

Comments
 (0)