Viet-Anh on Software Logo

What is: XLM?

SourceCross-lingual Language Model Pretraining
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

XLM is a Transformer based architecture that is pre-trained using one of three language modelling objectives:

  1. Causal Language Modeling - models the probability of a word given the previous words in a sentence.
  2. Masked Language Modeling - the masked language modeling objective of BERT.
  3. Translation Language Modeling - a (new) translation language modeling objective for improving cross-lingual pre-training.

The authors find that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models.