add elu and selu activations

[CarloLucibello]

No description provided.

[CarloLucibello]

this should be ready for merge after review

[cangumeli]

Shouldn't elu and selu be primitives if we add them to Knet?

[denizyuret]

You mean Autograd primitives? It depends. If they are defined generically using other primitives, no need. If they are type specific and/or there are performance concerns then yes.

…

On Wed, Feb 14, 2018 at 12:03 cangumeli ***@***.***> wrote: Shouldn't elu and selu be primitives if we add them to Knet? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#263 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvNpuzRtYTzzbZjSIeenINAodpCZKWRks5tUqFxgaJpZM4RuYY1> .

[cangumeli]

If elu and selu will be added after each weight layer similar to other activations, defining them as composite operations may bring performance and memory overhead in training. I think they should be implemented similar to how sigm, relu etc. are implemented.

[quaertym]

How about using elu here?

function selu(x)
    alpha = Float32(1.6732632)
    scale = Float32(1.0507009)
    return scale * elu(x, alpha)
end 

[rened]

Elu can be included in unary.jl by adding
("elu", "elu", "(xi>0?xi:exp(xi)-1)"),
and
(:elu, :eluback, :(ifelse(xi>0,xi,exp(xi)-1)), :(ifelse(yi>0,dyi,yi+1))),
below the respective relu lines.

[ozanarkancan]

This part turns to a cuda code snippet. So, I think it must be replaced with the following:

("eluback", "eluback", "(yi>0?xi:yi+1)")

[CarloLucibello]

I don't understand the comment. ifelse(yi>0,dyi,yi+1) is valid julia code and should be the right derivative

[ozanarkancan]

When I tried to build the package, it threw following errors:

cuda01.cu(395): error: identifier "dyi" is undefined
cuda01.cu(395): error: identifier "ifelse" is undefined
cuda01.cu(408): error: identifier "dyi" is undefined
cuda01.cu(408): error: identifier "ifelse" is undefined

I am able to build the package with the ("eluback", "eluback", "(yi>0?xi:yi+1)") code snippet.

[denizyuret]

Those snippets are not Julia code they are Cuda.

…

On Tue, Apr 17, 2018, 16:04 Ozan Arkan Can ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/broadcast.jl <#263 (comment)>: > @@ -32,6 +32,7 @@ broadcast_ops = [ # "fdim", ("invxback","invxback","(-xi*yi*yi)"), ("reluback","reluback","(yi>0?xi:0)"), + ("eluback", "eluback", "ifelse(yi>0,dyi,yi+1)"), When I tried to build the package, it threw following errors: cuda01.cu(395): error: identifier "dyi" is undefinedcuda01.cu(395): error: identifier "ifelse" is undefinedcuda01.cu(408): error: identifier "dyi" is undefinedcuda01.cu(408): error: identifier "ifelse" is undefined I am able to build the package with the ("eluback", "eluback", "(yi>0?xi:yi+1)") code snippet. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#263 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvNpl9qdn6jpfz7P8I2N58e4mDjfYtqks5tpeh3gaJpZM4RuYY1> .

[denizyuret]

Manually merged elu after fixing the faulty gradient: for negative values the derivative should be dyi*(yi+1).

[denizyuret]

Added selu as a cuda kernel for efficiency.

[denizyuret]

@CarloLucibello can you explain how this gives the intended result in alpha_dropout:
x = q*dropout(x .- alpha, p) .+ alpha #set dropped input to alpha