A **Sandwich Transformer** is a variant of a [Transformer](https://paperswithcode.com/method/transformer) that reorders sublayers in the architecture to achieve better performance. The reordering is based on the authors' analysis that models with more self-attention toward the bottom and more
feedforward sublayers toward the top tend to perform better in general.

**SongNet** is an auto-regressive [Transformer](https://paperswithcode.com/method/transformer)-based language model for rigid format text detection. Sets of symbols are tailor-designed to improve the modeling performance especially on format, rhyme, and sentence integrity. The attention mechanism is improved to impel the model to capture some future information on the format. A pre-training and fine-tuning framework is designed to further improve the generation quality.

SongNet

SongNet: Rigid Formats Controlled Text Generation

Sandwich Transformer

Improving Transformer Models by Reordering their Sublayers

Class activation guide is a module which uses weak localization information from the instrument activation maps to guide the verb and target recognition. 

Image source: [Nwoye et al.](https://arxiv.org/pdf/2007.05405v1.pdf)

Source	Improving Transformer Models by Reordering their Sublayers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com