Viet-Anh on Software Logo

What is: Lightweight Convolution?

SourcePay Less Attention with Lightweight and Dynamic Convolutions
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

LightConv is a type of depthwise convolution for sequential modelling which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax. Compared to self-attention, LightConv has a fixed context window and it determines the importance of context elements with a set of weights that do not change over time steps. LightConv computes the following for the ii-th element in the sequence and output channel cc:

LightConv(X,W_ceil(cHd),:,i,c)=DepthwiseConv(X,softmax(W_ceil(cHd),:),i,c)\text{LightConv}\left(X, W\_{\text{ceil}\left(\frac{cH}{d}\right),:}, i, c\right) = \text{DepthwiseConv}\left(X,\text{softmax}\left(W\_{\text{ceil}\left(\frac{cH}{d}\right),:}\right), i, c\right)