**AutoTinyBERT** is a an efficient [BERT](https://paperswithcode.com/method/bert) variant found through neural architecture search. Specifically, one-shot learning is used to obtain a big Super Pretrained Language Model (SuperPLM), where the objectives of pre-training or task-agnostic BERT distillation are used.  Then, given a specific latency constraint, an evolutionary algorithm is run on the SuperPLM to search optimal architectures. Finally, we extract the corresponding sub-models based on the optimal architectures and further train these models.

**ResNet-D** is a modification on the [ResNet](https://paperswithcode.com/method/resnet) architecture that utilises an [average pooling](https://paperswithcode.com/method/average-pooling) tweak for downsampling. The motivation is that in the unmodified ResNet, the 1 × 1 [convolution](https://paperswithcode.com/method/convolution) for the downsampling block ignores 3/4 of input feature maps, so this is modified so no information will be ignored

ResNet-D

Bag of Tricks for Image Classification with Convolutional Neural Networks

AutoTinyBERT

AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

A **Memory Network** provides a memory component that can be read from and written to with the inference capabilities of a neural network model. The motivation is that many neural networks lack a long-term memory component, and their existing memory component encoded by states and weights is too small and not compartmentalized enough to accurately remember facts from the past (RNNs for example, have difficult memorizing and doing tasks like copying). 

A memory network consists of a memory $\textbf{m}$ (an array of objects indexed by $\textbf{m}\_{i}$ and four potentially learned components:

- Input feature map $I$ - feature representation of the data input.
- Generalization $G$ - updates old memories given the new input.
- Output feature map $O$ - produces new feature map given $I$ and $G$.
- Response $R$ - converts output into the desired response. 

Given an input $x$ (e.g., an input character, word or sentence depending on the granularity chosen, an image or an audio signal) the flow of the model is as follows:

1. Convert $x$ to an internal feature representation $I\left(x\right)$.
2. Update memories $m\_{i}$ given the new input: $m\_{i} = G\left(m\_{i}, I\left(x\right), m\right)$, $\forall{i}$.
3. Compute output features $o$ given the new input and the memory: $o = O\left(I\left(x\right), m\right)$.
4. Finally, decode output features $o$ to give the final response: $r = R\left(o\right)$.

This process is applied at both train and test time, if there is a distinction between such phases, that
is, memories are also stored at test time, but the model parameters of $I$, $G$, $O$ and $R$ are not updated. Memory networks cover a wide class of possible implementations. The components $I$, $G$, $O$ and $R$ can potentially use any existing ideas from the machine learning literature.

Image Source: [Adrian Colyer](https://blog.acolyer.org/2016/03/10/memory-networks/)

Source	AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com