A **DenseNet** is a type of convolutional neural network that utilises [dense connections](https://paperswithcode.com/method/dense-connections) between layers, through [Dense Blocks](http://www.paperswithcode.com/method/dense-block), where we connect *all layers* (with matching feature-map sizes) directly with each other. To preserve the feed-forward nature, each layer obtains additional inputs from all preceding layers and passes on its own feature-maps to all subsequent layers.

Compress an ensemble of models into a single one by averaging their weights (under certain pre-conditions).

Soups

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

DenseNet

Densely Connected Convolutional Networks

**Reformer** is a [Transformer](https://paperswithcode.com/method/transformer) based architecture that seeks to make efficiency improvements. [Dot-product attention](https://paperswithcode.com/method/dot-product-attention) is replaced by one that uses locality-sensitive hashing, changing its complexity
from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, Reformers use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers.

Source	Densely Connected Convolutional Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com