**RealFormer** is a type of [Transformer](https://paperswithcode.com/methods/category/transformers) based on the idea of [residual](https://paperswithcode.com/method/residual-connection) attention. It adds skip edges to the backbone [Transformer](https://paperswithcode.com/method/transformer) to create multiple direct paths, one for each type of attention module. It adds no parameters or hyper-parameters. Specifically, RealFormer uses a Post-[LN](https://paperswithcode.com/method/layer-normalization) style Transformer as backbone and adds skip edges to connect [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention) modules in adjacent layers.

PGC-DGCNN provides a new definition of graph convolutional filter. It generalizes the most commonly adopted filter, adding an hyper-parameter controlling the distance of the considered neighborhood. The model extends graph convolutions, following an intuition derived from the well-known convolutional filters over multi-dimensional tensors. The methods involves a simple, efficient and effective way to introduce a hyper-parameter on graph convolutions that influences the filter size, i.e. its receptive field over the considered graph.

Description and image from: [On Filter Size in Graph Convolutional Networks](https://arxiv.org/pdf/1811.10435.pdf)

PGC-DGCNN

On Filter Size in Graph Convolutional Networks

RealFormer

RealFormer: Transformer Likes Residual Attention

A **Zero-padded Shortcut Connection** is a type of [residual connection](https://paperswithcode.com/method/residual-connection) used in the [PyramidNet](https://paperswithcode.com/method/pyramidnet) architecture. For PyramidNets, identity mapping alone cannot be used for a shortcut because the feature map dimension differs among individual residual units. Therefore, only a zero-padded shortcut or projection shortcut can be used for all the residual units. However,  a projection shortcut can hamper information propagation and lead to optimization problems, especially for very deep networks. On the other hand, the zero-padded shortcut avoids the overfitting problem because no additional parameters exist.

Source	RealFormer: Transformer Likes Residual Attention
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com