Viet-Anh on Software Logo

What is: Tokens-To-Token Vision Transformer?

SourceTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

T2T-ViT (Tokens-To-Token Vision Transformer) is a type of Vision Transformer which incorporates 1) a layerwise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be reduced; 2) an efficient backbone with a deep-narrow structure for vision transformer motivated by CNN architecture design after empirical study.