**BinaryBERT** is a [BERT](https://paperswithcode.com/method/bert)-variant that applies quantization in the form of weight binarization. Specifically, ternary weight splitting is proposed which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. To obtain BinaryBERT, we first train a half-sized [ternary BERT](https://paperswithcode.com/method/ternarybert) model, and then apply a [ternary weight splitting](https://paperswithcode.com/method/ternary-weight-splitting) operator to obtain the latent full-precision and quantized weights as the initialization of the full-sized BinaryBERT. We then fine-tune BinaryBERT for further refinement.

The **Linear Combination of Activations**, or **LinComb**, is a type of activation function that has trainable parameters and uses the linear combination of other activation functions.

$$LinComb(x) = \sum\limits_{i=0}^{n} w_i \mathcal{F}_i(x)$$

LinComb

Trainable Activations for Image Classification

BinaryBERT

BinaryBERT: Pushing the Limit of BERT Quantization

**Reversible Residual Blocks** are skip-connection blocks that learn *reversible* residual functions with reference to the layer inputs. It is proposed as part of the [RevNet](https://paperswithcode.com/method/revnet) CNN architecture. Units in each layer are partitioned into two groups, denoted $x\_{1}$ and $x\_{2}$; the authors find what works best is partitioning the channels. Each reversible block takes inputs $\left(x\_{1}, x\_{2}\right)$ and produces outputs $\left(y\_{1}, y\_{2}\right)$ according to the following additive coupling rules – inspired by the transformation in [NICE](https://paperswithcode.com/method/nice) (nonlinear independent components estimation) – and residual functions $F$ and $G$ analogous to those in standard [ResNets](https://paperswithcode.com/method/resnet):

$$y\_{1} = x\_{1} + F\left(x\_{2}\right)$$
$$y\_{2} = x\_{2} + G\left(y\_{1}\right)$$

Each layer’s activations can be reconstructed from the next layer’s activations as follows:

$$ x\_{2} = y\_{2} − G\left(y\_{1}\right)$$
$$ x\_{1} = y\_{1} − F\left(x\_{2}\right)$$

Source	BinaryBERT: Pushing the Limit of BERT Quantization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com