A **Zero-padded Shortcut Connection** is a type of [residual connection](https://paperswithcode.com/method/residual-connection) used in the [PyramidNet](https://paperswithcode.com/method/pyramidnet) architecture. For PyramidNets, identity mapping alone cannot be used for a shortcut because the feature map dimension differs among individual residual units. Therefore, only a zero-padded shortcut or projection shortcut can be used for all the residual units. However,  a projection shortcut can hamper information propagation and lead to optimization problems, especially for very deep networks. On the other hand, the zero-padded shortcut avoids the overfitting problem because no additional parameters exist.

**RealFormer** is a type of [Transformer](https://paperswithcode.com/methods/category/transformers) based on the idea of [residual](https://paperswithcode.com/method/residual-connection) attention. It adds skip edges to the backbone [Transformer](https://paperswithcode.com/method/transformer) to create multiple direct paths, one for each type of attention module. It adds no parameters or hyper-parameters. Specifically, RealFormer uses a Post-[LN](https://paperswithcode.com/method/layer-normalization) style Transformer as backbone and adds skip edges to connect [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention) modules in adjacent layers.

RealFormer

RealFormer: Transformer Likes Residual Attention

Zero-padded Shortcut Connection

Deep Pyramidal Residual Networks

**ShuffleNet** is a convolutional neural network designed specially for mobile devices with very limited computing power. The architecture utilizes two new operations, pointwise group [convolution](https://paperswithcode.com/method/convolution) and [channel shuffle](https://paperswithcode.com/method/channel-shuffle), to reduce computation cost while maintaining accuracy.

Source	Deep Pyramidal Residual Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com