PatchMerger is a module for Vision Transformers that decreases the number of tokens/patches passed onto each individual transformer encoder block whilst maintaining performance and reducing compute. PatchMerger takes linearly transforms an input of shape N patches × D dimensions through a learnable weight matrix of shape M output patches × D. This generates M scores, in which a Softmax function is applied for each score. The resulting output has a shape of M × N, which is multiplied to the original input to get an output of shape M × D.

Mathematically, $$Y = \text{softmax}({W^T}{X^T})X$$

Image and formula from: Renggli, C., Pinto, A. S., Houlsby, N., Mustafa, B., Puigcerver, J., & Riquelme, C. (2022). Learning to Merge Tokens in Vision Transformers. arXiv preprint arXiv:2202.12015.

**Euclidean Norm Regularization** is a regularization step used in [generative adversarial networks](https://paperswithcode.com/methods/category/generative-adversarial-networks), and is typically added to both the generator and discriminator losses:

$$ R\_{z} = w\_{r} \cdot ||\Delta{z}||^{2}\_{2} $$

where the scalar weight $w\_{r}$ is a parameter.

Image: [LOGAN](https://paperswithcode.com/method/logan)

Euclidean Norm Regularization

Deep Compressed Sensing

Patch Merger

Learning to Merge Tokens in Vision Transformers

**MixConv**, or **Mixed Depthwise Convolution**, is a type of [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) that naturally mixes up multiple kernel sizes in a single [convolution](https://paperswithcode.com/method/convolution). It is based on the insight that depthwise convolution applies a single kernel size to all channels, which MixConv overcomes by combining the benefits of multiple kernel sizes. It does this by partitioning channels into groups and applying a different kernel size to each group.

Source	Learning to Merge Tokens in Vision Transformers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com