**Meena** is a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. A seq2seq model is used with the Evolved [Transformer](https://paperswithcode.com/method/transformer) as the main architecture. The model is trained on multi-turn conversations where the input sequence is all turns of the context and the output sequence is the response.

[Transformer](https://paperswithcode.com/method/transformer) is a type of self-attention-based neural networks originally applied for NLP tasks. Recently, pure transformer-based models are proposed to solve computer vision problems. These visual transformers usually view an image as a sequence of patches while they ignore the intrinsic structure information inside each patch. In this paper, we propose a novel Transformer-iN-Transformer (TNT) model for modeling both patch-level and pixel-level representation. In each TNT block, an outer transformer block is utilized to process patch embeddings, and an inner transformer block extracts local features from pixel embeddings. The pixel-level feature is projected to the space of patch embedding by a linear transformation layer and then added into the patch. By stacking the TNT blocks, we build the TNT model for image recognition.

Image source: [Han et al.](https://arxiv.org/pdf/2103.00112v1.pdf)

Transformer in Transformer

Meena

Towards a Human-like Open-Domain Chatbot

**Mixture of Logistic Distributions (MoL)** is a type of output function, and an alternative to a [softmax](https://paperswithcode.com/method/softmax) layer. Discretized logistic mixture likelihood is used in [PixelCNN](https://paperswithcode.com/method/pixelcnn)++ and [WaveNet](https://paperswithcode.com/method/wavenet) to predict discrete values.

Image Credit: [Hao Gao](https://medium.com/@smallfishbigsea/an-explanation-of-discretized-logistic-mixture-likelihood-bdfe531751f0)

Source	Towards a Human-like Open-Domain Chatbot
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com