**Subformer** is a [Transformer](https://paperswithcode.com/method/transformer) that combines sandwich-style parameter sharing, which overcomes naive cross-layer parameter sharing in generative models, and self-attentive embedding factorization (SAFE). In SAFE, a small self-attention layer is used to reduce embedding parameter count.

**OverFeat** is a classic type of convolutional neural network architecture, employing [convolution](https://paperswithcode.com/method/convolution), pooling and fully connected layers. The Figure to the right shows the architectural details.

OverFeat

OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks

Subformer

Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers

**GeGLU** is an activation function which is a variant of [GLU](https://paperswithcode.com/method/glu). The definition is as follows:

$$ \text{GeGLU}\left(x, W, V, b, c\right) = \text{GELU}\left(xW + b\right) \otimes \left(xV + c\right) $$

Source	Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com