Viet-Anh on Software Logo

What is: Gated Positional Self-Attention?

SourceConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Gated Positional Self-Attention (GPSA) is a self-attention module for vision transformers, used in the ConViT architecture, that can be initialized as a convolutional layer -- helping a ViT learn inductive biases about locality.