**ProGAN**, or **Progressively Growing GAN**, is a generative adversarial network that utilises a progressively growing training approach. The idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses.

The **Maxout Unit** is a generalization of the [ReLU](https://paperswithcode.com/method/relu) and the [leaky ReLU](https://paperswithcode.com/method/leaky-relu) functions. It is a piecewise linear function that returns the maximum of the inputs, designed to be used in conjunction with [dropout](https://paperswithcode.com/method/dropout). Both ReLU and leaky ReLU are special cases of Maxout. 

$$f\left(x\right) = \max\left(w^{T}\_{1}x + b\_{1}, w^{T}\_{2}x + b\_{2}\right)$$

The main drawback of Maxout is that it is computationally expensive as it doubles the number of parameters for each neuron.

Maxout

Maxout Networks

ProGAN

Progressive Growing of GANs for Improved Quality, Stability, and Variation

**ALiBi**, or **Attention with Linear Biases**, is a [positioning method](https://paperswithcode.com/methods/category/position-embeddings) that allows [Transformer](https://paperswithcode.com/methods/category/transformers) language models to consume, at inference time, sequences which are longer than the ones they were trained on. 

ALiBi does this without using actual position embeddings. Instead, computing the attention between a certain key and query, ALiBi penalizes the attention value that that query can assign to the key depending on how far away the key and query are. So when a key and query are close by, the penalty is very low, and when they are far away, the penalty is very high. 

This method was motivated by the simple reasoning that words that are close-by matter much more than ones that are  far away.

This method is as fast as the sinusoidal or absolute embedding methods (the fastest positioning methods there are). It outperforms those methods and Rotary embeddings when evaluating sequences that are longer than the ones the model was trained on (this is known as extrapolation).

Source	Progressive Growing of GANs for Improved Quality, Stability, and Variation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com