**Cross-Covariance Attention**, or **XCA**, is an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) which operates along the feature dimension instead of the token dimension as in [conventional transformers](https://paperswithcode.com/methods/category/transformers).

Using the definitions of queries, keys and values from conventional attention, the cross-covariance attention function is defined as:

$$
\text { XC-Attention }(Q, K, V)=V \mathcal{A}_{\mathrm{XC}}(K, Q), \quad \mathcal{A}\_{\mathrm{XC}}(K, Q)=\operatorname{Softmax}\left(\hat{K}^{\top} \hat{Q} / \tau\right)
$$

where each output token embedding is a convex combination of the $d\_{v}$ features of its corresponding token embedding in $V$. The attention weights $\mathcal{A}$ are computed based on the cross-covariance matrix.

**AdaMod** is a stochastic optimizer that restricts adaptive learning rates with adaptive and momental upper bounds. The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks.


The weight updates are performed as:


$$ g\_{t} = \nabla{f}\_{t}\left(\theta\_{t-1}\right) $$

$$ m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)g\_{t} $$

$$ v\_{t} = \beta\_{2}v\_{t-1} + \left(1-\beta\_{2}\right)g\_{t}^{2} $$

$$ \hat{m}\_{t} = m\_{t} / \left(1 - \beta^{t}\_{1}\right)$$

$$ \hat{v}\_{t} = v\_{t} / \left(1 - \beta^{t}\_{2}\right)$$

$$ \eta\_{t} = \alpha\_{t} / \left(\sqrt{\hat{v}\_{t}} + \epsilon\right) $$

$$ s\_{t} = \beta\_{3}s\_{t-1} + (1-\beta\_{3})\eta\_{t} $$

$$ \hat{\eta}\_{t} = \text{min}\left(\eta\_{t}, s\_{t}\right) $$

$$ \theta\_{t} = \theta\_{t-1} - \hat{\eta}\_{t}\hat{m}\_{t} $$

AdaMod

An Adaptive and Momental Bound Method for Stochastic Learning

Cross-Covariance Attention

XCiT: Cross-Covariance Image Transformers

**VoiceFilter-Lite** is a single-channel source separation model that runs on the device to preserve only the speech signals from a target user, as part of a streaming speech recognition system. In this architecture, the voice filtering model operates as a frame-by-frame frontend signal processor to enhance the features consumed by the speech recognizer, without reconstructing audio signals from the features. The key contributions are (1) A system to perform speech separation directly on ASR input features; (2) An asymmetric loss function to penalize oversuppression during training, to make the model harmless under various acoustic environments, (3) An adaptive suppression strength mechanism to adapt to different noise conditions.

Source	XCiT: Cross-Covariance Image Transformers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com