Exit whenever the model is confident enough allowing early exiting from hidden layers

Sandwich Batch Normalization (**SaBN**) is a frustratingly easy improvement of [Batch Normalization](https://paperswithcode.com/method/batch-normalization) (BN) with only a few lines of code changes. SaBN is motivated by addressing the inherent *feature distribution heterogeneity* that one can be identified in many tasks, which can arise from data heterogeneity (multiple input domains) or model heterogeneity (dynamic architectures, model conditioning, etc.). Our SaBN factorizes the BN affine layer into one shared *sandwich affine* layer, cascaded by several parallel *independent affine* layers. We demonstrate the prevailing effectiveness of SaBN as a **drop-in replacement in four tasks**: *conditional image generation*, *[neural architecture search](https://paperswithcode.com/method/neural-architecture-search)* (NAS), *adversarial training*, and *arbitrary style transfer*. Leveraging SaBN immediately achieves better Inception Score and FID on CIFAR-10 and ImageNet conditional image generation with three state-of-the-art GANs; boosts the performance of a state-of-the-art weight-sharing NAS algorithm significantly on NAS-Bench-201; substantially improves the robust and standard accuracies for adversarial defense; and produces superior arbitrary stylized results.

SaBN

Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity

Early exiting

BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks

Dilated convolution with learnable spacings (DCLS) is a type of convolution that allows the spacings between the non-zero elements of the kernel to be learned during training. This makes it possible to increase the receptive field of the convolution without increasing the number of parameters, which can improve the performance of the network on tasks that require long-range dependencies.

A dilated convolution is a type of convolution that allows the kernel to be skipped over some of the input features. This is done by inserting zeros between the non-zero elements of the kernel. The effect of this is to increase the receptive field of the convolution without increasing the number of parameters.

DCLS takes this idea one step further by allowing the spacings between the non-zero elements of the kernel to be learned during training. This means that the network can learn to skip over different input features depending on the task at hand. This can be particularly helpful for tasks that require long-range dependencies, such as image segmentation and object detection.

DCLS has been shown to be effective for a variety of tasks, including image classification, object detection, and semantic segmentation. It is a promising new technique that has the potential to improve the performance of convolutional neural networks on a variety of tasks.

Source	BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com