**Self-Attention Network** (**SANet**) proposes two variations of self-attention used for image recognition: 1) pairwise self-attention which generalizes standard [dot-product attention](https://paperswithcode.com/method/dot-product-attention) and is fundamentally a set operator, and 2) patchwise self-attention which is strictly more powerful than [convolution](https://paperswithcode.com/method/convolution).

**Dense Connections**, or **Fully Connected Connections**, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output by a weight. This means there are $n\_{\text{inputs}}*n\_{\text{outputs}}$ parameters, which can lead to a lot of parameters for a sizeable network.

$$h\_{l} = g\left(\textbf{W}^{T}h\_{l-1}\right)$$

where $g$ is an activation function.

Image Source: Deep Learning by Goodfellow, Bengio and Courville

Dense Connections

SANet

Exploring Self-attention for Image Recognition

**CurricularFace**, or **Adaptive Curriculum Learning**, is a method for face recognition that embeds the idea of curriculum learning into the loss function to achieve a new training scheme. This training scheme mainly addresses easy samples in the early training stage and hard ones in the later stage. Specifically, CurricularFace adaptively adjusts the relative importance of easy and hard samples during different training stages.

Source	Exploring Self-attention for Image Recognition
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com