Viet-Anh on Software Logo

What is: DeLighT?

SourceDeLighT: Deep and Light-weight Transformer
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

DeLiGHT is a transformer architecture that delivers parameter efficiency improvements by (1) within each Transformer block using DExTra, a deep and light-weight transformation, allowing for the use of single-headed attention and bottleneck FFN layers and (2) across blocks using block-wise scaling, that allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output.