The **Shifted Rectified Linear Unit**, or **ShiLU**, is a modification of **[ReLU](https://paperswithcode.com/method/relu)** activation function that has trainable parameters.

$$ShiLU(x) = \alpha ReLU(x) + \beta$$

**LSH Attention**, or **Locality Sensitive Hashing Attention** is a replacement for [dot-product attention](https://paperswithcode.com/method/scaled) with one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. LSH refers to a family of functions (known as LSH families) to hash data points into buckets so that data points near each other are located in the same buckets with high probability, while data points far from each other are likely to be in different buckets. It was proposed as part of the [Reformer](https://paperswithcode.com/method/reformer) architecture.

LSH Attention

Reformer: The Efficient Transformer

ShiLU

Trainable Activations for Image Classification

**CodeT5** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based model for code understanding and generation based on the [T5 architecture](https://paperswithcode.com/method/t5). It utilizes an identifier-aware pre-training objective that considers the crucial token type information (identifiers) from code. Specifically, the denoising [Seq2Seq](https://paperswithcode.com/method/seq2seq) objective of T5 is extended with two identifier tagging and prediction tasks to enable the model to better leverage the token type information from programming languages, which are the identifiers assigned by developers. To improve the natural language-programming language alignment, a bimodal dual learning objective is used for a bidirectional conversion between natural language and programming language.

Source	Trainable Activations for Image Classification
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com