The **Pointer Sentinel-LSTM mixture model** is a type of recurrent neural network that combines the advantages of standard [softmax](https://paperswithcode.com/method/softmax) classifiers with those of a pointer component for effective and efficient language modeling. Rather than relying on the RNN hidden state to decide when to use the pointer, the model allows the pointer component itself to decide when to use the softmax vocabulary through a sentinel.

**Fully Convolutional Networks**, or **FCNs**, are an architecture used mainly for semantic segmentation. They employ solely locally connected layers, such as [convolution](https://paperswithcode.com/method/convolution), pooling and upsampling. Avoiding the use of dense layers means less parameters (making the networks faster to train). It also means an FCN can work for variable image sizes given all connections are local.

The network consists of a downsampling path, used to extract and interpret the context, and an upsampling path, which allows for localization. 

FCNs also employ skip connections to recover the fine-grained spatial information lost in the downsampling path.

Fully Convolutional Networks for Semantic Segmentation

Pointer Sentinel-LSTM

Pointer Sentinel Mixture Models

A **Neural Cache**, or a **Continuous Cache**, is a module for language modelling which stores previous hidden states in memory cells. They are then used as keys to retrieve their corresponding word, that is the next word. There is no transformation applied to the storage during writing and reading.

More formally it exploits the hidden representations $h\_{t}$ to define a probability distribution over the words in the cache. As
illustrated in the Figure, the cache stores pairs $\left(h\_{i}, x\_{i+1}\right)$ of a hidden representation, and the word which was generated based on this representation (the vector $h\_{i}$ encodes the history $x\_{i}, \dots, x\_{1}$). At time $t$, we then define a probability distribution over words stored in the cache based on the stored hidden representations and the current one $h\_{t}$ as:

$$ p\_{cache}\left(w | h\_{1\dots{t}}, x\_{1\dots{t}}\right) \propto \sum^{t-1}\_{i=1}\mathcal{1}\_{\text{set}\left(w=x\_{i+1}\right)} \exp\left(θ\_{h}>h\_{t}^{T}h\_{i}\right) $$

where the scalar $\theta$ is a parameter which controls the flatness of the distribution. When $\theta$ is equal to zero, the probability distribution over the history is uniform, and the model is equivalent to a unigram cache model.

Source	Pointer Sentinel Mixture Models
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com