Gated Attention Networks (GaAN) is a new architecture for learning on graphs. Unlike the traditional multi-head attention mechanism, which equally consumes all attention heads, GaAN uses a convolutional sub-network to control each attention head’s importance.

Image credit: [GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs](https://paperswithcode.com/paper/gaan-gated-attention-networks-for-learning-on)

A **Deformable Kernels** is a type of convolutional operator for deformation modeling. DKs learn free-form offsets on kernel coordinates to deform the original kernel space towards specific data modality, rather than recomposing data. This can directly adapt the effective receptive field (ERF) while leaving the receptive field untouched. They can be used as a drop-in replacement of rigid kernels. 

As shown in the Figure, for each input patch, a local DK first generates a group of kernel offsets $\{\Delta \mathcal{k}\}$ from input feature patch using the light-weight generator $\mathcal{G}$ (a 3$\times$3 [convolution](https://paperswithcode.com/method/convolution) of rigid kernel). Given the original kernel weights $\mathcal{W}$ and the offset group $\{\Delta \mathcal{k}\}$, DK samples a new set of kernel $\mathcal{W}'$ using a bilinear sampler $\mathcal{B}$. Finally, DK convolves the input feature map and the sampled kernels to complete the whole computation.

Deformable Kernel

Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation

GaAN

GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs

A **Ghost BottleNeck** is a skip connection block, similar to the basic [residual block](https://paperswithcode.com/method/residual-block) in [ResNet](https://paperswithcode.com/method/resnet) in which several convolutional layers and shortcuts are integrated, but stacks [Ghost Modules](https://paperswithcode.com/method/ghost-module) instead (two stacked Ghost modules). It was proposed as part of the [GhostNet](https://paperswithcode.com/method/ghostnet) CNN architecture.

The first Ghost module acts as an expansion layer increasing the number of channels. The ratio between the number of the output channels and that of the input is referred to as the *expansion ratio*. The second Ghost module reduces the number of channels to match the shortcut path. Then the shortcut is connected between the inputs and the outputs of these two Ghost modules. The [batch normalization](https://paperswithcode.com/method/batch-normalization) (BN) and [ReLU](https://paperswithcode.com/method/relu) nonlinearity are applied after each layer, except that ReLU is not used after the second Ghost module as suggested by [MobileNetV2](https://paperswithcode.com/method/mobilenetv2). The Ghost bottleneck described above is for stride=1. As for the case where stride=2, the shortcut path is implemented by a downsampling layer and a [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) with stride=2 is inserted between the two Ghost modules. In practice, the primary [convolution](https://paperswithcode.com/method/convolution) in Ghost module here is [pointwise convolution](https://paperswithcode.com/method/pointwise-convolution) for its efficiency.

Source	GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com