Viet-Anh on Software Logo

What is: Probabilistically Masked Language Model?

SourceProbabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Probabilistically Masked Language Model, or PMLM, is a type of language model that utilizes a probabilistic masking scheme, aiming to bridge the gap between masked and autoregressive language models. The basic idea behind the connection of two categories of models is similar to MADE by Germain et al (2015). PMLM is a masked language model with a probabilistic masking scheme, which defines the way sequences are masked by following a probabilistic distribution. The authors employ a simple uniform distribution of the masking ratio and name the model as u-PMLM.