A **Data-Efficient Image Transformer** is a type of [Vision Transformer](https://paperswithcode.com/method/vision-transformer) for image classification tasks. The model is trained using a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention.

**DBlock** is a residual based block used in the discriminator of the [GAN-TTS](https://paperswithcode.com/method/gan-tts) architecture. They are similar to the [GBlocks](https://paperswithcode.com/method/gblock) used in the generator, but without batch normalisation.

DBlock

High Fidelity Speech Synthesis with Adversarial Networks

DeiT

Training data-efficient image transformers & distillation through attention

TAM is designed to capture complex temporal relationships both  efficiently and  flexibly,
It adopts an adaptive kernel instead of self-attention to capture  global contextual information, with lower time complexity 
than GLTR.

TAM has two branches, a local branch and a global branch. Given the input feature map $X\in \mathbb{R}^{C\times T\times H\times W}$,  global spatial average pooling $\text{GAP}$ is first applied to the feature map to ensure TAM has a low computational cost. Then the local branch in TAM employs several 1D convolutions with ReLU nonlinearity across the temporal domain to produce location-sensitive importance maps for enhancing frame-wise features.
The local branch can be written as
\begin{align}
    s &= \sigma(\text{Conv1D}(\delta(\text{Conv1D}(\text{GAP}(X)))))
\end{align}
\begin{align}
    X^1 &= s X
\end{align}
Unlike the local branch, the global branch is location invariant and focuses on generating a channel-wise adaptive kernel based on global temporal information in each channel. For the $c$-th channel, the  kernel can be written as

\begin{align}
    \Theta_c = \text{Softmax}(\text{FC}_2(\delta(\text{FC}_1(\text{GAP}(X)_c)))) 
\end{align}

where $\Theta_c \in \mathbb{R}^{K}$ and $K$ is the adaptive kernel size. Finally, TAM  convolves the adaptive kernel $\Theta$ with $ X_\text{out}^1$:
\begin{align}
    Y = \Theta \otimes  X^1
\end{align}

With the help of the local branch and global branch,
TAM can capture the complex temporal structures in video and 
enhance per-frame features at low computational cost.
Due to its flexibility and lightweight design,
TAM can be added to any existing 2D CNNs.

Source	Training data-efficient image transformers & distillation through attention
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com