**T-Fixup** is an [initialization](https://paperswithcode.com/methods/category/initialization) method for [Transformers](https://paperswithcode.com/methods/category/transformers) that aims to remove the need for [layer normalization](https://paperswithcode.com/method/layer-normalization) and [warmup](https://paperswithcode.com/method/linear-warmup). The initialization procedure is as follows:

- Apply [Xavier initialization](https://paperswithcode.com/method/xavier-initialization) for all parameters excluding input embeddings. Use Gaussian initialization $\mathcal{N}\left(0, d^{-\frac{1}{2}}\right)$ for input embeddings where $d$ is the embedding dimension.
- Scale $\mathbf{v}\_{d}$ and $\mathbf{w}\_{d}$ matrices in each decoder [attention block](https://paperswithcode.com/method/multi-head-attention), weight matrices in each decoder [MLP block](https://paperswithcode.com/method/position-wise-feed-forward-layer) and input embeddings $\mathbf{x}$ and $\mathbf{y}$ in encoder and decoder by $(9 N)^{-\frac{1}{4}}$
- Scale $\mathbf{v}\_{e}$ and $\mathbf{w}\_{e}$ matrices in each encoder [attention block](https://paperswithcode.com/method/multi-head-attention) and weight matrices in each encoder [MLP block](https://paperswithcode.com/method/position-wise-feed-forward-layer) by $0.67 N^{-\frac{1}{4}}$

T-Fixup

Improving Transformer Optimization Through Better Initialization

**Contrastive Video Representation Learning**, or **CVRL**, is a self-supervised contrastive learning framework for learning spatiotemporal visual representations from unlabeled videos. Representations are learned using a contrastive loss, where two clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away. Data augmentations are designed involving spatial and temporal cues. Concretely, a [temporally consistent spatial augmentation](https://paperswithcode.com/method/temporally-consistent-spatial-augmentation#) method is used to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. A sampling-based temporal augmentation method is also used to avoid overly enforcing invariance on clips that are distant in time. 

End-to-end, from a raw video, we first sample a temporal interval from a monotonically decreasing distribution. The temporal interval represents the number of frames between the start points of two clips, and we sample two clips from a video according to this interval. Afterwards we apply a [temporally consistent spatial augmentation](https://paperswithcode.com/method/temporally-consistent-spatial-augmentation) to each of the clips and feed them into a 3D backbone with an MLP head. The contrastive loss is used to train the network to attract the clips from the same video and repel the clips from different videos in the embedding space.

CVRL

Spatiotemporal Contrastive Video Representation Learning

The Global Local Attention Module (GLAM) is an image model block that attends to the feature map's channels and spatial dimensions locally, and also attends to the feature map's channels and spatial dimensions globally. The locally attended feature maps, globally attended feature maps, and the original feature maps are then fused through a weighted sum (with learnable weights) to obtain the final feature map.

Paper:

Song, C. H., Han, H. J., & Avrithis, Y. (2022). All the attention you need: Global-local, spatial-channel attention for image retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2754-2763).

Global Local Attention Module

All the attention you need: Global-local, spatial-channel attention for image retrieval

The Triplet Entropy Loss (TEL) training method aims to leverage both the strengths of Cross Entropy Loss (CEL) and [Triplet loss](https://paperswithcode.com/method/triplet-loss) during the training process, assuming that it would lead to better generalization. The TEL method though does not contain a pre-training step, but trains simultaneously with both CEL and Triplet losses.

Triplet Entropy Loss

Triplet Entropy Loss: Improving The Generalisation of Short Speech Language Identification Systems

**PULSE** is a self-supervised photo upsampling algorithm. Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the downscaling loss, which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, the authors aim to restrict the search space to guarantee realistic outputs.

PULSE

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

Spatial pooling usually operates on a small region which limits its capability to capture long-range dependencies and focus on distant regions. To overcome this, Hou et al. proposed  strip pooling, a novel pooling method capable of encoding long-range context in either horizontal or vertical spatial domains.  

Strip pooling has two branches for horizontal  and vertical strip pooling. The horizontal strip pooling part first pools the input feature $F \in \mathcal{R}^{C \times H \times W}$ in the horizontal direction:
\begin{align}
y^1 = \text{GAP}^w (X) 
\end{align}
Then a 1D convolution with kernel size 3 is applied in $y$ to capture the relationship between different rows and channels. This is repeated $W$ times to make  the output $y_v$  consistent with the input shape:
\begin{align}
    y_h = \text{Expand}(\text{Conv1D}(y^1))
\end{align}
Vertical strip pooling is performed in a similar way. Finally, the outputs of the two branches are fused using element-wise summation to produce the attention map:
\begin{align}
s &= \sigma(Conv^{1\times 1}(y_{v} + y_{h}))
\end{align}
\begin{align}
Y &= s  X
\end{align}

The strip pooling module (SPM) is further developed in the mixed pooling module (MPM). Both consider  spatial  and channel relationships to overcome the locality of convolutional neural networks.  SPNet achieves  state-of-the-art results for several complex semantic segmentation benchmarks.

SPNet

Strip Pooling: Rethinking Spatial Pooling for Scene Parsing

OSCAR is a new learning method that uses object tags detected in images as anchor points to ease the learning of image-text alignment. The model take a triple as input (word-tag-region) and pre-trained with two losses (masked token loss over words and tags, and a contrastive loss between tags and others). OSCAR represents an image-text pair into semantic space via dictionary lookup. Object tags are used as anchor points to align image regions with word embeddings of pre-trained language models. The model is then fine-tuned for understanding and generation tasks.

OSCAR

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Non-parametric approximation of Q-values by storing all visited states and doing inference through k-Nearest Neighbors.

MFEC

Model-Free Episodic Control

**GRLIA** is an incident aggregation framework for online service systems based on graph representation learning over the cascading graph of cloud failures. A representation vector is learned for each unique type of incident in an unsupervised and unified manner, which is able to simultaneously encode the topological and temporal correlations among incidents.

GRLIA

Graph-based Incident Aggregation for Large-Scale Online Service Systems

**Computation Redistribution** is an [neural architecture search](https://paperswithcode.com/task/architecture-search) method for [face detection](https://paperswithcode.com/task/face-detection), which reallocates the computation between the backbone, neck and head of the model based on a predefined search methodology. Directly utilising the backbone of a classification network for scale-specific face detection can be sub-optimal. Therefore, [network structure search](https://paperswithcode.com/method/regnety) is used to reallocate the computation on the backbone, neck and head, under a wide range of flop regimes. The search method is applied to [RetinaNet](https://paperswithcode.com/method/retinanet), with [ResNet](https://paperswithcode.com/method/resnet) as backbone, [Path Aggregation Feature Pyramid Network](https://paperswithcode.com/method/pafpn) (PAFPN)  as the neck and stacked 3 × 3 [convolutional layers](https://paperswithcode.com/method/convolution) for the head. While the general structure is simple, the total number of possible networks in the search space is unwieldy. In the first step, the authors explore the reallocation of the computation within the backbone parts (i.e. stem, C2, C3, C4, and C5), while fixing the neck and head components. Based on the optimised computation distribution on the backbone they find, they further explore the reallocation of the computation across the backbone, neck and head.

Computation Redistribution

Sample and Computation Redistribution for Efficient Face Detection

**Tacotron** is an end-to-end generative text-to-speech model that takes a character sequence as input and outputs the corresponding spectrogram. The backbone of Tacotron is a seq2seq model with attention. The Figure depicts the model, which includes an encoder, an attention-based decoder, and a post-processing net. At a high-level, the model takes characters as input and produces spectrogram
frames, which are then converted to waveforms.

Tacotron

Tacotron: Towards End-to-End Speech Synthesis

**Video-Audio-Text Transformer**, or **VATT**, is a framework for learning multimodal representations from unlabeled data using [convolution](https://paperswithcode.com/method/convolution)-free [Transformer](https://paperswithcode.com/method/transformer) architectures. Specifically, it takes raw signals as inputs and extracts multidimensional representations that are rich enough to benefit a variety of downstream tasks. VATT borrows the exact architecture from [BERT](https://paperswithcode.com/method/bert) and [ViT](https://paperswithcode.com/method/vision-transformer) except the layer of tokenization and linear projection reserved for each modality separately. The design follows the same spirit as ViT that makes the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks.

VATT linearly projects each modality into a feature vector and feeds it into a Transformer encoder. A semantically hierarchical common space is defined to account for the granularity of different modalities and noise contrastive estimation is employed to train the model.

VATT

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

**Location-based Attention** is an attention mechanism in which the alignment scores are computed from solely the target hidden state $\mathbf{h}\_{t}$ as follows:

$$ \mathbf{a}\_{t} = \text{softmax}(\mathbf{W}\_{a}\mathbf{h}_{t}) $$

Location-based Attention

Effective Approaches to Attention-based Neural Machine Translation

**Multi-scale Progressive Fusion Network** (MSFPN) is a neural network representation for single image deraining. It aims to exploit the correlated information of rain streaks across scales for single image deraining. 

Specifically, we first generate the Gaussian pyramid rain images using Gaussian kernels to down-sample the original rain image in sequence. A coarse-fusion module (CFM) is designed to capture the global texture information from these multi-scale rain images through recurrent calculation (Conv-[LSTM](https://paperswithcode.com/method/lstm)), thus enabling the network to cooperatively represent the target rain streak using similar counterparts from global feature space. Meanwhile, the representation of the high-resolution pyramid layer is guided by previous outputs as well as all low-resolution pyramid layers. A finefusion module (FFM) is followed to further integrate these correlated information from different scales. By using the channel attention mechanism, the network not only discriminatively learns the scale-specific knowledge from all preceding pyramid layers, but also reduces the feature redundancy effectively. Moreover, multiple FFMs can be cascaded to form a progressive multi-scale fusion. Finally, a reconstruction module (RM) is appended to aggregate the coarse and fine rain information extracted respectively from CFM and FFM for learning the residual rain image, which is the approximation of real rain streak distribution.

MSPFN

Multi-Scale Progressive Fusion Network for Single Image Deraining

**BLANC** is an automatic estimation approach for document summary quality. The goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. BLANC achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document's text.

BLANC

Fill in the BLANC: Human-free quality estimation of document summaries

A self-supervised learning method to disentangle irregular (anomalous) motion from regular motion in unlabeled videos.

Motion Disentanglement

Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment

DBGAN is a method for graph representation learning. Instead of the widely used normal distribution assumption, the prior distribution of latent representation in DBGAN is estimated in a structure-aware way, which implicitly bridges the graph and feature spaces by prototype learning.

Source: [Distribution-induced Bidirectional Generative Adversarial Network for Graph Representation Learning](https://arxiv.org/abs/1912.01899)

DBGAN

**Weight Standardization** is a normalization technique that smooths the loss landscape by standardizing the weights in convolutional layers. Different from the previous normalization methods that focus on *activations*, WS considers the smoothing effects of *weights* more than just length-direction decoupling. Theoretically, WS reduces the Lipschitz constants of the loss and the gradients.
Hence, WS smooths the loss landscape and improves training.

In Weight Standardization, instead of directly optimizing the loss $\mathcal{L}$ on the original weights $\hat{W}$, we reparameterize the weights $\hat{W}$ as a function of $W$, i.e. $\hat{W}=\text{WS}(W)$, and optimize the loss $\mathcal{L}$ on $W$ by [SGD](https://paperswithcode.com/method/sgd):

$$
    \hat{W} = \Big[ \hat{W}\_{i,j}~\big|~ \hat{W}\_{i,j} = \dfrac{W\_{i,j} - \mu\_{W\_{i,\cdot}}}{\sigma\_{W\_{i,\cdot}+\epsilon}}\Big]
$$

$$
    y = \hat{W}*x
$$

where

$$
    \mu_{W\_{i,\cdot}} = \dfrac{1}{I}\sum\_{j=1}^{I}W\_{i, j},~~\sigma\_{W\_{i,\cdot}}=\sqrt{\dfrac{1}{I}\sum\_{i=1}^I(W\_{i,j} - \mu\_{W\_{i,\cdot}})^2}
$$

Similar to [Batch Normalization](https://paperswithcode.com/method/batch-normalization), WS controls the first and second moments of the weights of each output channel individually in convolutional layers. Note that many initialization methods also initialize the weights in some similar ways. Different from those methods, WS standardizes the weights in a differentiable way which aims to normalize gradients during back-propagation. Note that we do not have any affine transformation on $\hat{W}$. This is because we assume that normalization layers such as BN or [GN](https://paperswithcode.com/method/group-normalization) will normalize this convolutional layer again.

AI Glossary

T-Fixup

Contrastive Video Representation Learning

Global Local Attention Module

Triplet Entropy Loss

PULSE

Strip Pooling Network

OSCAR

Model-Free Episodic Control

GRLIA

Computation Redistribution

Tacotron

VATT

Location-based Attention

Multi-scale Progressive Fusion Network

BLANC

Self-Supervised Motion Disentanglement

Distribution-induced Bidirectional Generative Adversarial Network for Graph Representation Learning

Weight Standardization