Viet-Anh on Software Logo

What is: Single Headed Attention RNN?

SourceSingle Headed Attention RNN: Stop Thinking With Your Head
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

SHA-RNN, or Single Headed Attention RNN, is a recurrent neural network, and language model when combined with an embedding input and softmax classifier, based on a core LSTM component and a single-headed attention module. Other design choices include a Boom feedforward layer and the use of layer normalization. The guiding principles of the author were to ensure simplicity in the architecture and to keep computational costs bounded (the model was originally trained with a single GPU).