**RandAugment** is an automated data augmentation method. The search space for data augmentation has 2 interpretable hyperparameter $N$ and $M$.  $N$ is the number of augmentation transformations to apply sequentially, and $M$ is the magnitude for all the transformations. To reduce the parameter space but still maintain image diversity, learned policies and probabilities for applying each transformation are replaced with a parameter-free procedure of always selecting a transformation with uniform probability $\frac{1}{K}$. Here $K$ is the number of transformation options. So given $N$ transformations for a training image, RandAugment may thus express $KN$ potential policies.

Transformations applied include identity transformation, autoContrast, equalize, rotation, solarixation, colorjittering, posterizing, changing contrast, changing brightness, changing sharpness, shear-x, shear-y, translate-x, translate-y.

**Spectral Normalization** is a normalization technique used for generative adversarial networks, used to stabilize training of the discriminator. Spectral normalization has the convenient property that the Lipschitz constant is the only hyper-parameter to be tuned.

It controls the Lipschitz constant of the discriminator $f$ by constraining the spectral norm of each layer $g : \textbf{h}\_{in} \rightarrow \textbf{h}_{out}$. The Lipschitz norm $\Vert{g}\Vert\_{\text{Lip}}$ is equal to $\sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right)$, where $\sigma\left(a\right)$ is the spectral norm of the matrix $A$ ($L\_{2}$ matrix norm of $A$):

$$ \sigma\left(a\right) = \max\_{\textbf{h}:\textbf{h}\neq{0}}\frac{\Vert{A\textbf{h}}\Vert\_{2}}{\Vert\textbf{h}\Vert\_{2}} = \max\_{\Vert\textbf{h}\Vert\_{2}\leq{1}}{\Vert{A\textbf{h}}\Vert\_{2}} $$

which is equivalent to the largest singular value of $A$. Therefore for a [linear layer](https://paperswithcode.com/method/linear-layer) $g\left(\textbf{h}\right) = W\textbf{h}$ the norm is given by $\Vert{g}\Vert\_{\text{Lip}} = \sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right) = \sup\_{\textbf{h}}\sigma\left(W\right) = \sigma\left(W\right) $. Spectral normalization normalizes the spectral norm of the weight matrix $W$ so it satisfies the Lipschitz constraint $\sigma\left(W\right) = 1$:

$$ \bar{W}\_{\text{SN}}\left(W\right) = W / \sigma\left(W\right) $$

Spectral Normalization

Spectral Normalization for Generative Adversarial Networks

RandAugment

RandAugment: Practical automated data augmentation with a reduced search space

**LSH Attention**, or **Locality Sensitive Hashing Attention** is a replacement for [dot-product attention](https://paperswithcode.com/method/scaled) with one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. LSH refers to a family of functions (known as LSH families) to hash data points into buckets so that data points near each other are located in the same buckets with high probability, while data points far from each other are likely to be in different buckets. It was proposed as part of the [Reformer](https://paperswithcode.com/method/reformer) architecture.

Source	RandAugment: Practical automated data augmentation with a reduced search space
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com