**Shape Adaptor** is a novel resizing module for neural networks. It is a drop-in enhancement built on top of traditional resizing layers, such as pooling, bilinear sampling, and strided [convolution](https://paperswithcode.com/method/convolution). This module allows for a learnable shaping factor which differs from the traditional resizing layers that are fixed and deterministic.

Image Source: [Liu et al.](https://arxiv.org/pdf/2008.00892v2.pdf)

The **Affine Operator** is an affine transformation layer introduced in the [ResMLP](https://paperswithcode.com/method/resmlp) architecture. This replaces [layer normalization](https://paperswithcode.com/method/layer-normalization), as in [Transformer based networks](https://paperswithcode.com/methods/category/transformers), which is possible since in the ResMLP, there are no [self-attention layers](https://paperswithcode.com/method/scaled) which makes training more stable - hence allowing a more simple affine transformation.

The affine operator is defined as:

$$ \operatorname{Aff}_{\mathbf{\alpha}, \mathbf{\beta}}(\mathbf{x})=\operatorname{Diag}(\mathbf{\alpha}) \mathbf{x}+\mathbf{\beta} $$

where $\alpha$ and $\beta$ are learnable weight vectors. This operation only rescales and shifts the input element-wise. This operation has several advantages over other normalization operations: first, as opposed to Layer Normalization, it has no cost at inference time, since it can absorbed in the adjacent linear layer. Second, as opposed to [BatchNorm](https://paperswithcode.com/method/batch-normalization) and Layer Normalization, the Aff operator does not depend on batch statistics.

Affine Operator

ResMLP: Feedforward networks for image classification with data-efficient training

Shape Adaptor

Shape Adaptor: A Learnable Resizing Module

**GPipe** is a distributed model parallel method for neural networks. With GPipe, each model can be specified as a sequence of layers, and consecutive groups of layers can be partitioned into cells. Each cell is then placed on a separate accelerator. Based on this partitioned setup, batch splitting is applied. A mini-batch of training examples is split into smaller micro-batches, then the execution of each set of micro-batches is pipelined over cells. Synchronous mini-batch gradient descent is applied for training, where gradients are accumulated across all micro-batches in a mini-batch and applied at the end of a mini-batch.

Source	Shape Adaptor: A Learnable Resizing Module
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com