Viet-Anh on Software Logo

What is: Exponential Linear Squashing Activation?

SourceThe Quest for the Golden Activation Function
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

The Exponential Linear Squashing Activation Function, or ELiSH, is an activation function used for neural networks. It shares common properties with Swish, being made up of an ELU and a Sigmoid:

f(x)=x1+ex if x0f\left(x\right) = \frac{x}{1+e^{-x}} \text{ if } x \geq 0 f(x)=ex11+ex if x<0f\left(x\right) = \frac{e^{x} - 1}{1+e^{-x}} \text{ if } x < 0

The Sigmoid part of ELiSH improves information flow, while the linear parts solve issues of vanishing gradients.