A **Multiplicative LSTM (mLSTM)** is a  recurrent neural network architecture for sequence modelling that combines the long short-term memory ([LSTM](https://paperswithcode.com/method/lstm)) and multiplicative recurrent neural network ([mRNN](https://paperswithcode.com/method/mrnn)) architectures. The mRNN and LSTM architectures can be combined by adding connections from the mRNN’s intermediate state $m\_{t}$ to each gating units in the LSTM.

**Embedding Dropout** is equivalent to performing [dropout](https://paperswithcode.com/method/dropout) on the embedding matrix at a word level, where the dropout is broadcast across all the word vector’s embedding. The remaining non-dropped-out word embeddings are scaled by $\frac{1}{1-p\_{e}}$ where $p\_{e}$ is the probability of embedding dropout. As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a specific word will disappear within that pass, equivalent to performing [variational dropout](https://paperswithcode.com/method/variational-dropout) on the connection between the one-hot embedding and the embedding lookup.

Source: Merity et al, Regularizing and Optimizing [LSTM](https://paperswithcode.com/method/lstm) Language Models

Embedding Dropout

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

mLSTM

Multiplicative LSTM for sequence modelling

**Concatenation Affinity** is a type of affinity or self-similarity function between two points $\mathbb{x\_{i}}$ and $\mathbb{x\_{j}}$ that uses a concatenation function:

$$ f\left(\mathbb{x\_{i}}, \mathbb{x\_{j}}\right) = \text{ReLU}\left(\mathbb{w}^{T}\_{f}\left[\theta\left(\mathbb{x}\_{i}\right), \phi\left(\mathbb{x}\_{j}\right)\right]\right)$$

Here $\left[·, ·\right]$ denotes concatenation and $\mathbb{w}\_{f}$ is a weight vector that projects the concatenated vector to a scalar.

Source	Multiplicative LSTM for sequence modelling
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com