**Bort** is a parametric architectural variant of the [BERT](https://paperswithcode.com/method/bert) architecture. It extracts an optimal subset of architectural parameters for the BERT architecture through a [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) approach; in particular, a fully polynomial-time approximation scheme (FPTAS). This optimal subset - “Bort” - is demonstrably smaller, having an effective size of $5.5 \%$ the original BERT-large architecture, and $16\%$ of the net size. Bort is also able to be pretrained in $288$ GPU hours, which is $1.2\%$ less than the time required to pretrain the highest-performing BERT parametric architecture variant, RoBERTa-large ([RoBERTa](https://paperswithcode.com/method/roberta)), and about $33\%

**CAMoE** is a multi-stream Corpus Alignment network with single gate Mixture-of-Experts (MoE) for video-text retrieval. The CAMoE employs Mixture-of-Experts (MoE) to extract multi-perspective video representations, including action, entity, scene, etc., then align them with the corresponding part of the text. A [Dual Softmax Loss](https://paperswithcode.com/method/dual-softmax-loss) (DSL) is used to avoid the one-way optimum-match which occurs in previous contrastive methods. Introducing the intrinsic prior of each pair in a batch, DSL serves as a reviser to correct the similarity matrix and achieves the dual optimal match.

CAMoE

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Bort

Optimal Subarchitecture Extraction For BERT

**Gradient Normalization** is a normalization method for [Generative Adversarial Networks](https://paperswithcode.com/methods/category/generative-adversarial-networks) to tackle the training instability of generative adversarial networks caused by the sharp gradient space. Unlike existing work such as [gradient penalty](https://paperswithcode.com/method/wgan-gp-loss) and [spectral normalization](https://paperswithcode.com/method/spectral-normalization), the proposed GN only imposes a hard 1-Lipschitz constraint on the discriminator function, which increases the capacity of the network.

Source	Optimal Subarchitecture Extraction For BERT
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com