Viet-Anh on Software Logo

What is: Co-Scale Conv-attentional Image Transformer?

SourceCo-Scale Conv-Attentional Image Transformers
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Co-Scale Conv-Attentional Image Transformer (CoaT) is a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers' encoder branches at individual scales, while allowing representations learned at different scales to effectively communicate with each other. Second, the conv-attentional mechanism is designed by realizing a relative position embedding formulation in the factorized attention module with an efficient convolution-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities.