Viet-Anh on Software Logo

What is: Reformer?

SourceReformer: The Efficient Transformer
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Reformer is a Transformer based architecture that seeks to make efficiency improvements. Dot-product attention is replaced by one that uses locality-sensitive hashing, changing its complexity from O(L2L^2) to O(LlogLL\log L), where LL is the length of the sequence. Furthermore, Reformers use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of NN times, where NN is the number of layers.