**HetPipe** is a hybrid parallel method that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance.

**Res2Net** is an image model that employs a variation on bottleneck residual blocks. The motivation is to be able to represent features at multiple scales. This is achieved through a novel building block for CNNs that constructs hierarchical residual-like connections within one single [residual block](https://paperswithcode.com/method/residual-block).
This represents multi-scale features at a granular level and increases the range of receptive fields for each network layer.

Res2Net

Res2Net: A New Multi-scale Backbone Architecture

HetPipe

**TernaryBERT** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based model which ternarizes the weights of a pretrained [BERT](https://paperswithcode.com/method/bert) model to $\{-1,0,+1\}$, with different granularities for word embedding and weights in the Transformer layer. Instead of directly using knowledge distillation to compress a model, it is used to improve the performance of ternarized student model with the same size as the teacher model. In this way, we transfer the knowledge from the highly-accurate teacher model to the ternarized student model with smaller capacity.

Year	2020
Data Source	CC BY-SA - https://paperswithcode.com