Viet-Anh on Software Logo

What is: Compact Convolutional Transformers?

SourceEscaping the Big Data Paradigm with Compact Transformers
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Compact Convolutional Transformers utilize sequence pooling and replace the patch embedding with a convolutional embedding, allowing for better inductive bias and making positional embeddings optional. CCT achieves better accuracy than ViT-Lite (smaller ViTs) and increases the flexibility of the input parameters.