Viet-Anh on Software Logo

What is: Content-based Attention?

SourceNeural Turing Machines
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Content-based attention is an attention mechanism based on cosine similarity:

fatt(hi,s_j)=cos[h_i;s_j]f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) = \cos\left[\textbf{h}\_{i};\textbf{s}\_{j}\right]

It was utilised in Neural Turing Machines as part of the Addressing Mechanism.

We produce a normalized attention weighting by taking a softmax over these attention alignment scores.