**MixConv**, or **Mixed Depthwise Convolution**, is a type of [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) that naturally mixes up multiple kernel sizes in a single [convolution](https://paperswithcode.com/method/convolution). It is based on the insight that depthwise convolution applies a single kernel size to all channels, which MixConv overcomes by combining the benefits of multiple kernel sizes. It does this by partitioning channels into groups and applying a different kernel size to each group.

PatchMerger is a module for Vision Transformers that decreases the number of tokens/patches passed onto each individual transformer encoder block whilst maintaining performance and reducing compute. PatchMerger takes linearly transforms an input of shape N patches × D dimensions through a learnable weight matrix of shape M output patches × D. This generates M scores, in which a Softmax function is applied for each score. The resulting output has a shape of M × N, which is multiplied to the original input to get an output of shape M × D.

Mathematically, $$Y = \text{softmax}({W^T}{X^T})X$$

Image and formula from: Renggli, C., Pinto, A. S., Houlsby, N., Mustafa, B., Puigcerver, J., & Riquelme, C. (2022). Learning to Merge Tokens in Vision Transformers. arXiv preprint arXiv:2202.12015.

Patch Merger

Learning to Merge Tokens in Vision Transformers

MixConv

MixConv: Mixed Depthwise Convolutional Kernels

A method to convert a Tree Ensemble model into a Rule list. This makes the AI model more transparent.

Source	MixConv: Mixed Depthwise Convolutional Kernels
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com