Viet-Anh on Software Logo

What is: FT-Transformer?

SourceRevisiting Deep Learning Models for Tabular Data
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

FT-Transformer (Feature Tokenizer + Transformer) is a simple adaptation of the Transformer architecture for the tabular domain. The model (Feature Tokenizer component) transforms all features (categorical and numerical) to tokens and runs a stack of Transformer layers over the tokens, so every Transformer layer operates on the feature level of one object. (This model is similar to AutoInt). In the Transformer component, the [CLS] token is appended to TT. Then LL Transformer layers are applied. PreNorm is used for easier optimization and good performance. The final representation of the [CLS] token is used for prediction.