**Mode Normalization** extends normalization to more than a single mean and variance, allowing for detection of modes of data on-the-fly, jointly normalizing samples that share common features. It first assigns samples in a mini-batch to different modes via a gating network, and then normalizes each sample with estimators for its corresponding mode.

**modReLU** is an activation that is a modification of a [ReLU](https://paperswithcode.com/method/relu). It is a pointwise nonlinearity, $\sigma\_{modReLU}\left(z\right) : C \rightarrow C$, which affects only the absolute value of a complex number, defined as:

$$ \sigma\_{modReLU}\left(z\right) = \left(|z| + b\right)\frac{z}{|z|} \text{ if } |z| + b \geq 0 $$
$$ \sigma\_{modReLU}\left(z\right) = 0 \text{ if } |z| + b \leq 0 $$

where $b \in \mathbb{R}$ is a bias parameter of the nonlinearity. For a $n\_{h}$ dimensional hidden space we learn $n\_{h}$ nonlinearity bias parameters, one per dimension.

modReLU

Unitary Evolution Recurrent Neural Networks

Mode Normalization

An **Recursive Feature Pyramid (RFP)** builds on top of the Feature Pyramid Networks ([FPN](https://paperswithcode.com/method/fpn)) by incorporating extra feedback connections from the FPN layers into the bottom-up backbone layers. Unrolling the recursive structure to a sequential implementation, we obtain a backbone for object detector that looks at the images twice or more. Similar to the cascaded detector heads in [Cascade R-CNN](https://paperswithcode.com/method/cascade-r-cnn) trained with more selective examples, an RFP recursively enhances FPN to generate increasingly powerful representations. Resembling Deeply-Supervised Nets, the feedback connections bring the features that directly receive gradients from the detector heads back to the low levels of the bottom-up backbone to speed up training and boost performance.

Source	Mode Normalization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Mode Normalization?

Viet-Anh on Software