The **Sinkhorn Transformer** is a type of [transformer](https://paperswithcode.com/method/transformer) that uses [Sparse Sinkhorn Attention](https://paperswithcode.com/method/sparse-sinkhorn-attention) as a building block. This component is a plug-in replacement for dense fully-connected attention (as well as local attention, and sparse attention alternatives), and allows for reduced memory complexity as well as sparse attention.

**Factorized Random Synthesized Attention**, introduced with the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture, is similar to [factorized dense synthesized attention](https://paperswithcode.com/method/factorized-dense-synthesized-attention) but for random synthesizers. Letting $R$ being a randomly initialized matrix, we factorize $R$ into low rank matrices $R\_{1}, R\_{2} \in \mathbb{R}^{l\text{ x}k}$ in the attention function:

$$ Y = \text{Softmax}\left(R\_{1}R\_{2}^{T}\right)G\left(X\right) . $$

Here $G\left(.\right)$ is a parameterized function that is equivalent to $V$ in [Scaled Dot-Product Attention](https://paperswithcode.com/method/scaled).

For each head, the factorization reduces the parameter costs from $l^{2}$ to $2\left(lk\right)$ where
$k << l$ and hence helps prevent overfitting. In practice, we use a small value of $k = 8$.

The basic idea of a  Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-specific alignment that works well globally across many samples.

Factorized Random Synthesized Attention

Synthesizer: Rethinking Self-Attention in Transformer Models

Sinkhorn Transformer

Sparse Sinkhorn Attention

**PixelShuffle** is an operation used in super-resolution models to implement efficient sub-pixel convolutions with a stride of $1/r$. Specifically it rearranges elements in a tensor of shape $(\*, C \times r^2, H, W)$ to a tensor of shape $(\*, C, H \times r, W \times r)$.

Image Source: [Remote Sensing Single-Image Resolution Improvement Using A Deep Gradient-Aware Network with Image-Specific Enhancement](https://www.researchgate.net/figure/The-pixel-shuffle-layer-transforms-feature-maps-from-the-LR-domain-to-the-HR-image_fig3_339531308)

Source	Sparse Sinkhorn Attention
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com