**T5**, or **Text-to-Text Transfer Transformer**, is a [Transformer](https://paperswithcode.com/method/transformer) based architecture that uses a text-to-text approach. Every task – including translation, question answering, and classification – is cast as feeding the model text as input and training it to generate some target text. This allows for the use of the same model, loss function, hyperparameters, etc. across our diverse set of tasks. The changes compared to [BERT](https://paperswithcode.com/method/bert) include:

- adding a *causal* decoder to the bidirectional architecture.
- replacing the fill-in-the-blank cloze task with a mix of alternative pre-training tasks.

**Activation Normalization** is a type of normalization used for flow-based generative models; specifically it was introduced in the [GLOW](https://paperswithcode.com/method/glow) architecture. An ActNorm layer performs an affine transformation of the activations using a scale and bias parameter per channel, similar to [batch normalization](https://paperswithcode.com/method/batch-normalization). These parameters are initialized such that the post-actnorm activations per-channel have zero mean and unit variance given an initial minibatch of data. This is a form of data dependent initilization. After initialization, the scale and bias are treated as regular trainable parameters that are independent of the data.

Activation Normalization

Glow: Generative Flow with Invertible 1x1 Convolutions

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ALP-GMM is is an algorithm that learns to generate a learning curriculum for black box reinforcement learning agents, whereby it sequentially samples parameters controlling a stochastic procedural generation of tasks or environments.

Source	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com