An implementation of model & data parallel [GPT3-like](https://paperswithcode.com/method/gpt-3) models using the [mesh-tensorflow](https://github.com/tensorflow/mesh) library.

Source: [EleutherAI/GPT-Neo](https://github.com/EleutherAI/gpt-neo)

**ShuffleNet V2 Downsampling Block** is a block for spatial downsampling used in the [ShuffleNet V2](https://paperswithcode.com/method/shufflenet-v2) architecture. Unlike the regular [ShuffleNet](https://paperswithcode.com/method/shufflenet) V2 block, the channel split operator is removed so the number of output channels is doubled.

ShuffleNet V2 Downsampling Block

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

GPT-Neo

VL-BERT is pre-trained on a large-scale image-captions dataset together with text-only corpus. The input to the model are either words from the input sentences or regions-of-interest (RoI) from input images. It can be fine-tuned to fit most visual-linguistic downstream tasks. Its backbone is a multi-layer bidirectional Transformer encoder, modified to accommodate visual contents, and new type of visual feature embedding to the input feature embeddings. VL-BERT takes both visual and linguistic elements as input, represented as RoIs in images and subwords in input sentences. Four different types of embeddings are used to represent each input: token embedding, visual feature embedding, segment embedding, and sequence position embedding. VL-BERT is pre-trained using Conceptual Captions and text-only datasets. Two pre-training tasks are used: masked language modeling with visual clues, and masked RoI classification with linguistic clues.

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com