An oscillatory function defined as $x \cdot cos(x)$ that reports better performance than Sigmoid, Mish, Swish, and ReLU on several benchmarks.

Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks

Weight Reset is an implicit regularization procedure that periodically resets a randomly selected portion of layer weights during the training process, according to predefined probability distributions.

To delineate the Weight Reset procedure, a straightforward formulation is introduced. Assume $\mathcal{B}(p)$ as a multivariate Bernoulli distribution with parameter $p$, and let's propose that $\mathcal{D}$ is an arbitrary distribution used for initializing model weights. At specified intervals (after a certain number of training iterations, except for the last one), a random portion of the weights $W={w^l}$ from selected layers in the neural network undergoes a reset utilizing the following method:
$$
\tilde{w}^l = w^l\cdot (1-m) + \xi\cdot m,
$$
where $\cdot$ operation is an element-wise hadamar type multiplication, $w^l$ are current weights for layer $l$, $\tilde{w}^l$ are reset weights for this layer, $m \sim \mathcal{B}(p^l)$ is a resetting mask, $p^l$ is a resetting rate for a layer $l$, $\xi \sim \mathcal{D}$ are new random weights.

Evidence has indicated that Weight Reset can compete with, and in some instances, surpass traditional regularization techniques.

Given the observable effects of the Weight Reset technique on an increasing number of weights in a model, there's a plausible hypothesis suggesting its potential association with the Double Descent phenomenon.

Weights Reset

The Weights Reset Technique for Deep Neural Networks Implicit Regularization

**YellowFin** is a learning rate and momentum tuner motivated by robustness properties and analysis of quadratic objectives. It stems from a known but obscure fact: the momentum operator's spectral radius is constant in a large subset of the hyperparameter space. For quadratic objectives, the optimizer tunes both the learning rate and the momentum to keep the hyperparameters within a region in which the convergence rate is a constant rate equal to the root momentum. This notion is extended empirically to non-convex objectives. On every iteration, YellowFin optimizes the hyperparameters to minimize a local quadratic optimization.

YellowFin

YellowFin and the Art of Momentum Tuning

**Global and Sliding Window Attention** is an attention pattern for attention-based models. It is motivated by the fact that non-sparse attention in the original [Transformer](https://paperswithcode.com/method/transformer) formulation has a [self-attention component](https://paperswithcode.com/method/scaled) with $O\left(n^{2}\right)$ time and memory complexity where $n$ is the input sequence length and thus, is not efficient to scale to long inputs. 

Since [windowed](https://paperswithcode.com/method/sliding-window-attention) and [dilated](https://paperswithcode.com/method/dilated-sliding-window-attention) attention patterns are not flexible enough to learn task-specific representations, the authors of the [Longformer](https://paperswithcode.com/method/longformer) add “global attention” on few pre-selected input locations. This attention is operation symmetric: that is, a token with a global attention attends to all tokens across the sequence, and all tokens in the sequence attend to it. The Figure to the right shows an example of a sliding window attention with global attention at a few tokens at custom locations. For the example of classification, global attention is used for the [CLS] token, while in the example of Question Answering, global attention is provided on all question tokens.

Global and Sliding Window Attention

Longformer: The Long-Document Transformer

**WaveRNN** is a single-layer recurrent neural network for audio generation that is designed efficiently predict 16-bit raw audio samples.

The overall computation in the WaveRNN is as follows (biases omitted for brevity):

$$ \mathbf{x}\_{t} = \left[\mathbf{c}\_{t−1},\mathbf{f}\_{t−1}, \mathbf{c}\_{t}\right] $$

$$ \mathbf{u}\_{t} = \sigma\left(\mathbf{R}\_{u}\mathbf{h}\_{t-1} + \mathbf{I}^{*}\_{u}\mathbf{x}\_{t}\right) $$

$$ \mathbf{r}\_{t} = \sigma\left(\mathbf{R}\_{r}\mathbf{h}\_{t-1} + \mathbf{I}^{*}\_{r}\mathbf{x}\_{t}\right) $$

$$ \mathbf{e}\_{t} = \tau\left(\mathbf{r}\_{t} \odot \left(\mathbf{R}\_{e}\mathbf{h}\_{t-1}\right) + \mathbf{I}^{*}\_{e}\mathbf{x}\_{t} \right) $$

$$ \mathbf{h}\_{t} = \mathbf{u}\_{t} \cdot \mathbf{h}\_{t-1} + \left(1-\mathbf{u}\_{t}\right) \cdot \mathbf{e}\_{t} $$

$$ \mathbf{y}\_{c}, \mathbf{y}\_{f} = \text{split}\left(\mathbf{h}\_{t}\right) $$

$$ P\left(\mathbf{c}\_{t}\right) = \text{softmax}\left(\mathbf{O}\_{2}\text{relu}\left(\mathbf{O}\_{1}\mathbf{y}\_{c}\right)\right) $$

$$ P\left(\mathbf{f}\_{t}\right) = \text{softmax}\left(\mathbf{O}\_{4}\text{relu}\left(\mathbf{O}\_{3}\mathbf{y}\_{f}\right)\right) $$

where the $*$ indicates a masked matrix whereby the last coarse input $\mathbf{c}\_{t}$ is only connected to the fine part of the states $\mathbf{u}\_{t}$, $\mathbf{r}\_{t}$, $\mathbf{e}\_{t}$ and $\mathbf{h}\_{t}$ and thus only affects the fine output $\mathbf{y}\_{f}$. The coarse and fine parts $\mathbf{c}\_{t}$ and $\mathbf{f}\_{t}$ are encoded as scalars in $\left[0, 255\right]$ and scaled to the interval $\left[−1, 1\right]$. The matrix $\mathbf{R}$ formed from the matrices $\mathbf{R}\_{u}$, $\mathbf{R}\_{r}$, $\mathbf{R}\_{e}$ is computed as a single matrix-vector product to produce the contributions to all three gates $\mathbf{u}\_{t}$, $mathbf{r}\_{t}$ and $\mathbf{e}\_{t}$ (a variant of the [GRU cell](https://paperswithcode.com/method/gru). $\sigma$ and $\tau$ are the standard sigmoid and tanh non-linearities.

Each part feeds into a [softmax](https://paperswithcode.com/method/softmax) layer over the corresponding 8 bits and the prediction of the 8 fine bits is conditioned on the 8 coarse bits. The resulting Dual Softmax layer allows for efficient prediction of 16-bit samples using two small output spaces (2 8 values each) instead of a single large output space (with 2 16 values).

WaveRNN

Efficient Neural Audio Synthesis

**U-Net** is an architecture for semantic segmentation. It consists of a contracting path and an expansive path. The contracting path follows the typical architecture of a convolutional network. It consists of the repeated application of two 3x3 convolutions (unpadded convolutions), each followed by a rectified linear unit ([ReLU](https://paperswithcode.com/method/relu)) and a 2x2 [max pooling](https://paperswithcode.com/method/max-pooling) operation with stride 2 for downsampling. At each downsampling step we double the number of feature channels. Every step in the expansive path consists of an upsampling of the feature map followed by a 2x2 [convolution](https://paperswithcode.com/method/convolution) (“up-convolution”) that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU. The cropping is necessary due to the loss of border pixels in every convolution. At the final layer a [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) is used to map each 64-component feature vector to the desired number of classes. In total the network has 23 convolutional layers.

[Original MATLAB Code](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/u-net-release-2015-10-02.tar.gz)

U-Net

U-Net: Convolutional Networks for Biomedical Image Segmentation

In the field of fusing multi-spectral and panchromatic images (Pan-sharpening), the impressive effectiveness of deep neural networks has been recently employed to overcome the drawbacks of traditional linear models and boost the fusing accuracy. However, to the best of our knowledge, existing research works are mainly based on simple and flat networks with relatively shallow architecture, which severely limited their performances. In this paper, the concept of residual learning has been introduced to form a very deep convolutional neural network to make a full use of the high non-linearity of deep learning models. By both quantitative and visual assessments on a large number of high quality multi-spectral images from various sources, it has been supported that our proposed model is superior to all mainstream algorithms included in the comparison, and achieved the highest spatial-spectral unified accuracy.

DRPNN

Boosting the accuracy of multi-spectral image pan-sharpening by learning a deep residual network

**MobileNet** is a type of convolutional neural network designed for mobile and embedded vision applications. They are based on a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks that can have low latency for mobile and embedded devices.

MobileNetV1

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Smish is an activation function defined as $f(x)=x\cdot \text{tanh}(\ln(1+\sigma(x)))$ where $\sigma(x)$ denotes the sigmoid function. A parameterized version was also described in the form $f(x)=\alpha x\cdot \text{tanh}(\ln(1+\sigma(\beta x)))$.

Paper: Smish: A Novel Activation Function for Deep Learning Methods

Source: https://www.mdpi.com/2079-9292/11/4/540

Smish

**SimCLR** is a framework for contrastive learning of visual representations. It learns representations by maximizing agreement between differently augmented views of the same data example via a contrastive loss in the latent space. It consists of:

- A stochastic data augmentation module that transforms any given data example randomly resulting in two correlated views of the same example, denoted $\mathbf{\tilde{x}\_{i}}$ and $\mathbf{\tilde{x}\_{j}}$, which is considered a positive pair. SimCLR sequentially applies three simple augmentations: random cropping followed by resize back to the original size, random color distortions, and [random Gaussian blur](https://paperswithcode.com/method/random-gaussian-blur). The authors find random crop and color distortion is crucial to achieve good performance.

- A neural network base encoder $f\left(·\right)$ that extracts representation vectors from augmented data examples. The framework allows various choices of the network architecture without any constraints. The authors opt for simplicity and adopt [ResNet](https://paperswithcode.com/method/resnet) to obtain $h\_{i} = f\left(\mathbf{\tilde{x}}\_{i}\right) = \text{ResNet}\left(\mathbf{\tilde{x}}\_{i}\right)$ where $h\_{i} \in \mathbb{R}^{d}$ is the output after the [average pooling](https://paperswithcode.com/method/average-pooling) layer.

- A small neural network projection head $g\left(·\right)$ that maps representations to the space where contrastive loss is applied. Authors use a MLP with one hidden layer to obtain $z\_{i} = g\left(h\_{i}\right) = W^{(2)}\sigma\left(W^{(1)}h\_{i}\right)$ where $\sigma$ is a [ReLU](https://paperswithcode.com/method/relu) nonlinearity. The authors find it beneficial to define the contrastive loss on $z\_{i}$’s rather than $h\_{i}$’s.

- A contrastive loss function defined for a contrastive prediction task. Given a set {$\mathbf{\tilde{x}}\_{k}$} including a positive pair of examples $\mathbf{\tilde{x}}\_{i}$ and $\mathbf{\tilde{x}\_{j}}$ , the contrastive prediction task aims to identify $\mathbf{\tilde{x}}\_{j}$ in {$\mathbf{\tilde{x}}\_{k}$}$\_{k\neq{i}}$ for a given $\mathbf{\tilde{x}}\_{i}$.

A minibatch of $N$ examples is randomly sampled and the contrastive prediction task is defined on pairs of augmented examples derived from the minibatch, resulting in $2N$ data points. Negative examples are not sampled explicitly. Instead, given a positive pair, the other $2(N − 1)$ augmented examples within a minibatch are treated as negative examples. A [NT-Xent](https://paperswithcode.com/method/nt-xent) (the normalized
temperature-scaled cross entropy loss) loss function is used (see components).

SimCLR

A Simple Framework for Contrastive Learning of Visual Representations

**RandomHorizontalFlip** is a type of image data augmentation which horizontally flips a given image with a given probability.

Image Credit: [Apache MXNet](https://mxnet.apache.org/versions/1.5.0/tutorials/gluon/data_augmentation.html)

Random Horizontal Flip

MetaFormer is a general architecture abstracted from Transformers by not specifying the token mixer.

MetaFormer

MetaFormer Is Actually What You Need for Vision

**Height-driven Attention Network**, or **HANet**, is a general add-on module for improving semantic segmentation for urban-scene images. It emphasizes informative features or classes selectively according to the vertical position of a pixel. The pixel-wise class distributions are significantly different from each other among horizontally segmented sections in the urban-scene images. Likewise, urban-scene images have their own distinct characteristics, but most semantic segmentation networks do not reflect such unique attributes in the architecture. The proposed network architecture incorporates the capability exploiting the attributes to handle the urban scene dataset effectively.

HANet

Cars Can't Fly up in the Sky: Improving Urban-Scene Segmentation via Height-driven Attention Networks

**Balanced Feature Pyramid** is a feature pyramid module. It differs from approaches like [FPNs](https://paperswithcode.com/method/fpn) that integrate multi-level features using lateral connections. Instead the BFP strengthens the multi-level features using the same deeply integrated balanced semantic features. The pipeline is shown in the Figure to the right. It consists of four steps, rescaling, integrating, refining and strengthening.

Features at resolution level $l$ are denoted as $C\_{l}$. The number of multi-level features is denoted as $L$. The indexes of involved lowest and highest levels are denoted as $l\_{min}$ and $l\_{max}$. In the Figure, $C\_{2}$ has the highest resolution. To integrate multi-level features and preserve their semantic hierarchy at the same time, we first resize the multi-level features {$C\_{2}, C\_{3}, C\_{4}, C\_{5}$} to an intermediate size, i.e., the same size as $C\_{4}$, with interpolation and max-pooling respectively. Once the features are rescaled, the balanced semantic features are obtained by simple averaging as:

$$ C = \frac{1}{L}\sum^{l\_{max}}\_{l=l\_{min}}C\_{l} $$

The obtained features are then rescaled using the same but reverse procedure to strengthen the original features. Each resolution obtains equal information from others in this procedure. Note that this procedure does not contain any parameter. The authors observe improvement with this nonparametric method, proving the effectiveness of the information flow. 

The balanced semantic features can be further refined to be more discriminative. The authors found both the refinements with convolutions directly and the non-local module work well. But the
non-local module works in a more stable way. Therefore, embedded Gaussian non-local attention is utilized as default. The refining step helps us enhance the integrated features and further improve the results.

With this method, features from low-level to high-level are aggregated at the same time. The outputs
{$P\_{2}, P\_{3}, P\_{4}, P\_{5}$} are used for object detection following the same pipeline in FPN.

Balanced Feature Pyramid

Libra R-CNN: Towards Balanced Learning for Object Detection

A **Dilated Causal Convolution** is a [causal convolution](https://paperswithcode.com/method/causal-convolution) where the filter is applied over an area larger than its length by skipping input values with a certain step. A dilated causal [convolution](https://paperswithcode.com/method/convolution) effectively allows the network to have very large receptive fields with just a few layers.

Dilated Causal Convolution

WaveNet: A Generative Model for Raw Audio

Parameter tuning method for neural network models with adaptive activation functions.

DSPT

Adaptive hybrid activation function for deep neural networks

**AdaShift** is a type of adaptive stochastic optimizer that decorrelates $v\_{t}$ and $g\_{t}$ in [Adam](https://paperswithcode.com/method/adam) by temporal shifting, i.e., using temporally shifted gradient $g\_{t−n}$ to calculate $v\_{t}$. The authors argue that an inappropriate correlation between gradient $g\_{t}$ and the second-moment term $v\_{t}$ exists in Adam, which results in a large gradient being likely to have a small step size while a small gradient may have a large step size. The authors argue that such biased step sizes are the fundamental cause of non-convergence of Adam.

The AdaShift updates, based on the idea of temporal independence between gradients, are as follows:

$$ g\_{t} = \nabla{f\_{t}}\left(\theta\_{t}\right) $$

$$ m\_{t} = \sum^{n-1}\_{i=0}\beta^{i}\_{1}g\_{t-i}/\sum^{n-1}\_{i=0}\beta^{i}\_{1} $$

Then for $i=1$ to $M$:

$$ v\_{t}\left[i\right] = \beta\_{2}v\_{t-1}\left[i\right] + \left(1-\beta\_{2}\right)\phi\left(g^{2}\_{t-n}\left[i\right]\right) $$

$$ \theta\_{t}\left[i\right] = \theta\_{t-1}\left[i\right] - \alpha\_{t}/\sqrt{v\_{t}\left[i\right]}\cdot{m\_{t}\left[i\right]} $$

AdaShift

AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods

**SFAM**, or **Scale-wise Feature Aggregation Module**, is a feature extraction block from the [M2Det](https://paperswithcode.com/method/m2det) architecture. It aims to aggregate the multi-level multi-scale features generated by [Thinned U-Shaped Modules](https://paperswithcode.com/method/tum) into a multi-level feature pyramid. 

The first stage of SFAM is to concatenate features of the equivalent scale together along the channel dimension. The aggregated feature pyramid can be presented as $\mathbf{X} =[\mathbf{X}\_1,\mathbf{X}\_2,\dots,\mathbf{X}\_i]$, where $\mathbf{X}\_i = \text{Concat}(\mathbf{x}\_i^1,\mathbf{x}\_i^2,\dots,\mathbf{x}\_i^L) \in \mathbb{R}^{W\_{i}\times H\_{i}\times C}$ refers to the features of the $i$-th largest scale. Here, each scale in the aggregated pyramid contains features from multi-level depths. 

However, simple concatenation operations are not adaptive enough. In the second stage, we introduce a channel-wise attention module to encourage features to focus on channels that they benefit most. Following Squeeze-and-Excitation, we use [global average pooling](https://paperswithcode.com/method/global-average-pooling) to generate channel-wise statistics $\mathbf{z} \in \mathbb{R}^C$ at the squeeze step. And to fully capture channel-wise dependencies, the following excitation step learns the attention mechanism via two fully connected layers:

$$
\mathbf{s} = \mathbf{F}\_{ex}(\mathbf{z},\mathbf{W}) = \sigma(\mathbf{W}\_{2} \delta(\mathbf{W}\_{1}\mathbf{z})),
$$

where $\sigma$ refers to the [ReLU](https://paperswithcode.com/method/relu) function, $\delta$ refers to the sigmoid function, $\mathbf{W}\_{1} \in \mathbb{R}^{\frac{C}{r}\times C}$ , $\mathbf{W}\_{2} \in \mathbb{R}^{C\times \frac{C}{r}}$, r is the reduction ratio ($r=16$ in our experiments). The final output is obtained by reweighting the input $\mathbf{X}$ with activation $\mathbf{s}$:

$$
\tilde{\mathbf{X}}_i^c = \mathbf{F}\_{scale}(\mathbf{X}\_i^c,s_c) = s_c \cdot \mathbf{X}_i^c,
$$

where $\tilde{\mathbf{X}\_i} = [\tilde{\mathbf{X}}\_i^1,\tilde{\mathbf{X}}\_i^2,...,\tilde{\mathbf{X}}\_i^C]$, each of the features is enhanced or weakened by the rescaling operation.

AI Glossary

Growing Cosine Unit