Cluster-GCN is a novel GCN algorithm that is suitable for SGD-based training by exploiting the graph clustering structure. Cluster-GCN works as the following: at each step, it samples a block of nodes that associate with a dense subgraph identified by a graph clustering algorithm, and restricts the neighborhood search within this subgraph. This simple but effective strategy leads to significantly improved memory and computational efficiency while being able to achieve comparable test accuracy with previous algorithms.

Description and image from: [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://arxiv.org/pdf/1905.07953.pdf)

**Feature Intertwiner** is an object detection module that leverages the features from a more reliable set to help guide the feature learning of another less reliable set. The mutual learning process helps two sets to have closer distance within the cluster in each class. The intertwiner is applied on the object detection task, where a historical buffer is proposed to address the sample missing problem during one mini-batch and the optimal transport (OT) theory is introduced to enforce the similarity among the two sets.

Feature Intertwiner

Feature Intertwiner for Object Detection

Cluster-GCN

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

**gMLP** is an [MLP](https://paperswithcode.com/methods/category/feedforward-networks)-based alternative to [Transformers](https://paperswithcode.com/methods/category/vision-transformer) without [self-attention](https://paperswithcode.com/method/scaled), which simply consists of channel projections and spatial projections with static parameterization. It is built out of basic MLP layers with gating. The model consists of a stack of $L$ blocks with identical size and structure. Let $X \in \mathbb{R}^{n \times d}$ be the token representations with sequence length $n$ and dimension $d$. Each block is defined as:

$$
Z=\sigma(X U), \quad \tilde{Z}=s(Z), \quad Y=\tilde{Z} V
$$

where $\sigma$ is an activation function such as [GeLU](https://paperswithcode.com/method/gelu). $U$ and $V$ define linear projections along the channel dimension - the same as those in the FFNs of Transformers (e.g., their shapes are $768 \times 3072$ and $3072 \times 768$ for $\text{BERT}_{\text {base }}$).

A key ingredient is $s(\cdot)$, a layer which captures spatial interactions. When $s$ is an identity mapping, the above transformation degenerates to a regular FFN, where individual tokens are processed independently without any cross-token communication. One of the major focuses is therefore to design a good $s$ capable of capturing complex spatial interactions across tokens. This leads to the use of a [Spatial Gating Unit](https://www.paperswithcode.com/method/spatial-gating-unit) which involves a modified linear gating.

The overall block layout is inspired by [inverted bottlenecks](https://paperswithcode.com/method/inverted-residual-block), which define $s(\cdot)$ as a [spatial depthwise convolution](https://paperswithcode.com/method/depthwise-separable-convolution). Note, unlike Transformers, gMLP does not require position embeddings because such information will be captured in $s(\cdot)$.

Source	Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com