**LSH Attention**, or **Locality Sensitive Hashing Attention** is a replacement for [dot-product attention](https://paperswithcode.com/method/scaled) with one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. LSH refers to a family of functions (known as LSH families) to hash data points into buckets so that data points near each other are located in the same buckets with high probability, while data points far from each other are likely to be in different buckets. It was proposed as part of the [Reformer](https://paperswithcode.com/method/reformer) architecture.

**RandAugment** is an automated data augmentation method. The search space for data augmentation has 2 interpretable hyperparameter $N$ and $M$.  $N$ is the number of augmentation transformations to apply sequentially, and $M$ is the magnitude for all the transformations. To reduce the parameter space but still maintain image diversity, learned policies and probabilities for applying each transformation are replaced with a parameter-free procedure of always selecting a transformation with uniform probability $\frac{1}{K}$. Here $K$ is the number of transformation options. So given $N$ transformations for a training image, RandAugment may thus express $KN$ potential policies.

Transformations applied include identity transformation, autoContrast, equalize, rotation, solarixation, colorjittering, posterizing, changing contrast, changing brightness, changing sharpness, shear-x, shear-y, translate-x, translate-y.

RandAugment

RandAugment: Practical automated data augmentation with a reduced search space

LSH Attention

Reformer: The Efficient Transformer

The **Shifted Rectified Linear Unit**, or **ShiLU**, is a modification of **[ReLU](https://paperswithcode.com/method/relu)** activation function that has trainable parameters.

$$ShiLU(x) = \alpha ReLU(x) + \beta$$

Source	Reformer: The Efficient Transformer
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com