Viet-Anh on Software Logo

What is: Boom Layer?

SourceSingle Headed Attention RNN: Stop Thinking With Your Head
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

A Boom Layer is a type of feedforward layer that is closely related to the feedforward layers used in Transformers. The layer takes a vector of the form vRHv \in \mathbb{R}^{H} and uses a matrix multiplication with a GeLU activation to produce a vector uRN×Hu \in \mathbb{R}^{N\times{H}}. We then break uu into NN vectors and sum those together, producing wRHw \in \mathbb{R}^{H}. This minimizes computation and removes an entire matrix of parameters compared to traditional down-projection layers.

The Figure to the right shows the Boom Layer used in the context of SHA-RNN from the original paper.