A **HaloNet** is a self-attention based model for efficient image classification. It relies on a local self-attention architecture that efficiently maps to existing hardware with haloing. The formulation breaks translational equivariance, but the authors observe that it improves  throughput and accuracies over the centered local self-attention used in regular self-attention. The approach also utilises a strided self-attentive downsampling operation for multi-scale feature extraction.

**Contrastive Predictive Coding v2 (CPC v2)** is a self-supervised learning approach that builds upon the original [CPC](https://paperswithcode.com/method/contrastive-predictive-coding) with several improvements. These improvements include:

- **Model capacity** - The third residual stack of [ResNet](https://paperswithcode.com/method/resnet)-101 (originally containing 23 blocks, 1024-dimensional feature maps, and 256-dimensional bottleneck layers), is converted to use 46 blocks, with 4096-dimensional feature maps and 512-dimensional bottleneck layers: ResNet-161.

- **Layer Normalization** - The authors find CPC with [batch normalization](https://paperswithcode.com/method/batch-normalization) harms downstream performance. They hypothesize this is due to batch normalization allowing large models to find a trivial solution to CPC: it introduces a dependency between patches (through the batch statistics) that can be exploited to bypass the constraints on the receptive field. They replace batch normalization with [layer normalization](https://paperswithcode.com/method/layer-normalization).

- **Predicting lengths and directions** - patches are predicted with contexts from both directions rather than just spatially underneath.

- **Patch-based Augmentation** - Utilising "color dropping" which randomly drops two of the three color channels in each patch, as well as random horizontal flips.


Consistent with prior results, this new architecture delivers better performance regardless of

CPC v2

Data-Efficient Image Recognition with Contrastive Predictive Coding

HaloNet

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

**CornerNet-Squeeze Hourglass** is a convolutional neural network and object detection backbone used in the [CornerNet-Squeeze](https://paperswithcode.com/method/cornernet-squeeze) object detector. It uses a modified [hourglass module](https://paperswithcode.com/method/hourglass-module) that makes use of a [fire module](https://paperswithcode.com/method/fire-module): containing 1x1 convolutions and depthwise convolutions.

Source	Scaling Local Self-Attention for Parameter Efficient Visual Backbones
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com