**XLSR** is a multilingual speech recognition model built on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining. A shared quantization module over feature encoder representations produces multilingual quantized speech units whose embeddings are then used as targets for a [Transformer](https://paperswithcode.com/method/transformer) trained by contrastive learning. The model learns to share discrete tokens across languages, creating bridges across languages.

**Variational Trace Distance Estimation**, or **VTDE**, is a variational algorithm for trace norm estimation that only involves one ancillary qubit. Notably, the cost function in VTDE gathers information from a single-qubit observable and thus could avoid the barren plateau issue with logarithmic depth parameterized circuits.

VTDE

Variational Quantum Algorithms for Trace Distance and Fidelity Estimation

XLSR

Unsupervised Cross-lingual Representation Learning for Speech Recognition

**Universal Language Model Fine-tuning**, or **ULMFiT**, is an architecture and transfer learning method that can be applied to NLP tasks. It involves a 3-layer [AWD-LSTM](https://paperswithcode.com/method/awd-lstm) architecture for its representations. The training consists of three steps: 1) general language model pre-training on a Wikipedia-based text, 2) fine-tuning the language model on a target task, and 3) fine-tuning the classifier on the target task.

As different layers capture different types of information, they are fine-tuned to different extents using [discriminative fine-tuning](https://paperswithcode.com/method/discriminative-fine-tuning). Training is performed using [Slanted triangular learning rates](https://paperswithcode.com/method/slanted-triangular-learning-rates) (STLR), a learning rate scheduling strategy that first linearly increases the learning rate and then linearly decays it.

Fine-tuning the target classifier is achieved in ULMFiT using gradual unfreezing. Rather than fine-tuning all layers at once, which risks catastrophic forgetting, ULMFiT gradually unfreezes the model starting from the last layer (i.e., closest to the output) as this contains the least general knowledge. First the last layer is unfrozen and all unfrozen layers are fine-tuned for one epoch. Then the next group of frozen layers is unfrozen and fine-tuned and repeat, until all layers are fine-tuned until convergence at the last iteration.

Source	Unsupervised Cross-lingual Representation Learning for Speech Recognition
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com