**Mish** is an activation function for neural networks which can be defined as:

$$ f\left(x\right) = x\cdot\tanh{\text{softplus}\left(x\right)}$$

where

$$\text{softplus}\left(x\right) = \ln\left(1+e^{x}\right)$$

(Compare with functionally similar previously proposed activation functions such as the [GELU](https://paperswithcode.com/method/silu) $x\Phi(x)$ and the [SiLU](https://paperswithcode.com/method/silu) $x\sigma(x)$.)

**ControlVAE** is a [variational autoencoder](https://paperswithcode.com/method/vae) (VAE) framework that combines the automatic control theory with the basic VAE to stabilize the KL-divergence of VAE models to a specified value. It leverages a non-linear PI controller, a variant of the proportional-integral-derivative (PID) control, to dynamically tune the weight of the KL-divergence term in the evidence lower bound (ELBO) using the output KL-divergence as feedback. This allows for control of the KL-divergence to a desired value (set point), which is effective in avoiding posterior collapse and learning disentangled representations.

ControlVAE

ControlVAE: Tuning, Analytical Properties, and Performance Analysis

Mish

Mish: A Self Regularized Non-Monotonic Activation Function

**ZCA Whitening** is an image preprocessing method that leads to a transformation of data such that the covariance matrix $\Sigma$ is the identity matrix, leading to decorrelated features.

Image Source: [Alex Krizhevsky](http://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf)

Source	Mish: A Self Regularized Non-Monotonic Activation Function
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Mish?

Viet-Anh on Software