**Probabilistically Masked Language Model**, or **PMLM**, is a type of language model that utilizes a probabilistic masking scheme, aiming to bridge the gap between masked and autoregressive language models. The basic idea behind the connection of two categories of models is similar to MADE by Germain et al (2015). PMLM is a masked language model with a probabilistic masking scheme, which defines the way sequences are masked by following a probabilistic distribution. The authors employ a simple uniform distribution of the masking ratio and name the model as u-PMLM.

**Tacotron 2** is a neural network architecture for speech synthesis directly from text. It consists of two components:

- a recurrent sequence-to-sequence feature prediction network with
attention which predicts a sequence of mel spectrogram frames from
an input character sequence
- a modified version of [WaveNet](https://paperswithcode.com/method/wavenet) which generates time-domain waveform samples conditioned on the
predicted mel spectrogram frames

In contrast to the original [Tacotron](https://paperswithcode.com/method/tacotron), Tacotron 2 uses simpler building blocks, using vanilla [LSTM](https://paperswithcode.com/method/lstm) and convolutional layers in the encoder and decoder instead of [CBHG](https://paperswithcode.com/method/cbhg) stacks and [GRU](https://paperswithcode.com/method/gru) recurrent layers. Tacotron 2 does not use a “reduction factor”, i.e., each decoder step corresponds to a single spectrogram frame. Location-sensitive attention is used instead of [additive attention](https://paperswithcode.com/method/additive-attention).

Tacotron 2

Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions

PMLM

Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order

In human action recognition, 
each type of action  generally only depends 
on a few specific kinematic joints. Furthermore, over time, multiple actions may be performed.
Motivated by these observations, Song et al. proposed 
a joint spatial and temporal attention network based on LSTM, to adaptively find discriminative features and keyframes. 
Its main attention-related components are a spatial attention sub-network, to select important regions, and a temporal attention sub-network, to select key frames. The spatial attention sub-network can be written as:
\begin{align}
    s_{t} &= U_{s}\tanh(W_{xs}X_{t} + W_{hs}h_{t-1}^{s} + b_{si}) + b_{so}
\end{align}
\begin{align}
    \alpha_{t} &= \text{Softmax}(s_{t})
\end{align}
\begin{align}
    Y_{t} &= \alpha_{t}  X_{t} 
\end{align}
where $X_{t}$ is the input feature at time $t$, $U_{s}$, $W_{hs}$, $b_{si}$, and $b_{so}$ are learnable parameters, and $h_{t-1}^{s}$ is the hidden state at step $t-1$. Note that use of the hidden state $h$ means  the attention process takes  temporal relationships into consideration.

The temporal attention sub-network is similar to the spatial branch and produces its attention map using:
\begin{align}
    \beta_{t} = \delta(W_{xp}X_{t} + W_{hp}h_{t-1}^{p} + b_{p}). 
\end{align}
It adopts a ReLU function instead of a normalization function for ease of optimization. It also uses a regularized objective function to improve  convergence.

Overall, this paper presents a joint spatiotemporal attention method
to focus on important joints and keyframes, 
with excellent results on the action recognition task.

Source	Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com