**Off-Diagonal Orthogonal Regularization** is a modified form of [orthogonal regularization](https://paperswithcode.com/method/orthogonal-regularization) originally used in [BigGAN](https://paperswithcode.com/method/biggan). The original orthogonal regularization is known to be limiting so the authors explore several variants designed to relax the constraint while still imparting the desired smoothness to the models. They opt for a modification where they remove diagonal terms from the regularization, and aim to minimize the pairwise cosine similarity between filters but does not constrain their norm:

$$ R\_{\beta}\left(W\right) = \beta|| W^{T}W \odot \left(\mathbf{1}-I\right) ||^{2}\_{F} $$

where $\mathbf{1}$ denotes a matrix with all elements set to 1. The authors sweep $\beta$ values and select $10^{−4}$.

PoolFormer is instantiated from MetaFormer by specifying the token mixer as extremely simple operator, pooling. PoolFormer is utilized as a tool to verify MetaFormer hypothesis "MetaFormer is actually what you need" (vs "Attention is all you need").

PoolFormer

MetaFormer Is Actually What You Need for Vision

Off-Diagonal Orthogonal Regularization

Large Scale GAN Training for High Fidelity Natural Image Synthesis

**CornerNet-Saccade** is an extension of [CornerNet](https://paperswithcode.com/method/cornernet) with an attention mechanism similar to saccades in human vision. It starts with a downsized full image and generates an attention map, which is then zoomed in on and processed further by the model. This differs from the original CornerNet in that it is applied fully convolutionally across multiple scales.

Source	Large Scale GAN Training for High Fidelity Natural Image Synthesis
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com