**Decorrelated Batch Normalization (DBN)** 
is a normalization technique which not just centers and scales activations but whitens them. ZCA whitening instead of [PCA](https://paperswithcode.com/method/pca) whitening is employed since PCA whitening causes a problem called *stochastic axis swapping*, which is detrimental to learning.

**ALiBi**, or **Attention with Linear Biases**, is a [positioning method](https://paperswithcode.com/methods/category/position-embeddings) that allows [Transformer](https://paperswithcode.com/methods/category/transformers) language models to consume, at inference time, sequences which are longer than the ones they were trained on. 

ALiBi does this without using actual position embeddings. Instead, computing the attention between a certain key and query, ALiBi penalizes the attention value that that query can assign to the key depending on how far away the key and query are. So when a key and query are close by, the penalty is very low, and when they are far away, the penalty is very high. 

This method was motivated by the simple reasoning that words that are close-by matter much more than ones that are  far away.

This method is as fast as the sinusoidal or absolute embedding methods (the fastest positioning methods there are). It outperforms those methods and Rotary embeddings when evaluating sequences that are longer than the ones the model was trained on (this is known as extrapolation).

ALiBi

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Decorrelated Batch Normalization

**Feature Fusion Module v1** is a feature fusion module from the [M2Det](https://paperswithcode.com/method/m2det) object detection model, and feature fusion modules are crucial for constructing the final multi-level feature pyramid. They use [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) layers to compress the channels of the input features and use concatenation operation to aggregate these feature map. FFMv1 takes two feature maps with different scales in backbone as input, it adopts one upsample operation to rescale the deep features to the same scale before the concatenation operation.

Source	Decorrelated Batch Normalization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com