**Pretext-Invariant Representation Learning (PIRL, pronounced as “pearl”)** learns invariant representations based on pretext tasks. PIRL is used with a commonly used pretext task that involves solving [jigsaw](https://paperswithcode.com/method/jigsaw) puzzles. Specifically, PIRL constructs image representations that are similar to the representation of transformed versions of the same image and different from the representations of other images.

\begin{equation}
QA\left( x \right) = \sigma\left( f\left( x \right)^{1x1} \right) + x \end{equation}

Quick Attention takes in the feature map as an input WxHxC (Width x Height x Channels) and creates two instances of the input feature map then it performs the 1x1xC convolution on the first instance and calculates the sigmoid activations after that it is added with the second instance to generate the final attention map as output which is of same dimensions as of input.

Quick Attention

HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images

PIRL

Self-Supervised Learning of Pretext-Invariant Representations

**Sparsemax** is a type of activation/output function similar to the traditional [softmax](https://paperswithcode.com/method/softmax), but able to output sparse probabilities. 

$$ \text{sparsemax}\left(z\right) = \arg\_{p∈\Delta^{K−1}}\min||\mathbf{p} - \mathbf{z}||^{2} $$

Source	Self-Supervised Learning of Pretext-Invariant Representations
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com