Viet-Anh on Software Logo

What is: Mish?

SourceMish: A Self Regularized Non-Monotonic Activation Function
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Mish is an activation function for neural networks which can be defined as:

f(x)=xtanhsoftplus(x) f\left(x\right) = x\cdot\tanh{\text{softplus}\left(x\right)}

where

softplus(x)=ln(1+ex)\text{softplus}\left(x\right) = \ln\left(1+e^{x}\right)

(Compare with functionally similar previously proposed activation functions such as the GELU xΦ(x)x\Phi(x) and the SiLU xσ(x)x\sigma(x).)