Perhaps the most widely used approach to synthesizing new examples is called the Synthetic Minority Oversampling Technique, or SMOTE for short. This technique was described by Nitesh Chawla, et al. in their 2002 paper named for the technique titled “SMOTE: Synthetic Minority Over-sampling Technique.”

SMOTE works by selecting examples that are close in the feature space, drawing a line between the examples in the feature space and drawing a new sample at a point along that line.

Diffusion models applied to latent spaces, which are normally built with (Variational) Autoencoders.

Latent Diffusion Model

High-Resolution Image Synthesis with Latent Diffusion Models

SMOTE

SMOTE: Synthetic Minority Over-sampling Technique

A **Gated Linear Unit**, or **GLU** computes:

$$ \text{GLU}\left(a, b\right) = a\otimes \sigma\left(b\right) $$

It is used in natural language processing architectures, for example the [Gated CNN](https://paperswithcode.com/method/gated-convolution-network), because here $b$ is the gate that control what information from $a$ is passed up to the following layer. Intuitively, for a language modeling task, the gating mechanism allows selection of words or features that are important for predicting the next word. The GLU also has non-linear capabilities, but has a linear path for the gradient so diminishes the vanishing gradient problem.

Source	SMOTE: Synthetic Minority Over-sampling Technique
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com