**ESPNet** is a convolutional neural network for semantic segmentation of high resolution images under resource constraints. ESPNet is based on a convolutional module, efficient spatial pyramid ([ESP](https://paperswithcode.com/method/esp)), which is efficient in terms of computation, memory, and power.

SENet pioneered channel attention. The core of SENet is a squeeze-and-excitation (SE) block which is used to collect global information, capture channel-wise relationships and improve representation ability.
SE blocks are divided into two parts, a squeeze module and an excitation module. Global spatial information is collected in the squeeze module by global average pooling. The excitation module captures channel-wise relationships and outputs an attention vector by using fully-connected layers and non-linear layers (ReLU and sigmoid). Then, each channel of the input feature is scaled by multiplying the corresponding element in the attention vector. Overall, a squeeze-and-excitation block $F_\text{se}$ (with parameter $\theta$) which takes $X$ as input and outputs $Y$ can be formulated 
as:
\begin{align}
    s = F_\text{se}(X, \theta) & = \sigma (W_{2} \delta (W_{1}\text{GAP}(X)))
\end{align}
\begin{align}
    Y = sX
\end{align}

Channel attention

Squeeze-and-Excitation Networks

ESPNet

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

**Wide&Deep** jointly trains wide linear models and deep neural networks to combine the benefits of memorization and generalization for real-world recommender systems. In summary, the wide component is a generalized linear model. The deep component is a feed-forward neural network. The deep and wide components are combined using a weighted sum of their output log odds as the prediction. This is then fed to a logistic loss function for joint training, which is done by back-propagating the gradients from the output to both the wide and deep part of the model simultaneously using mini-batch stochastic optimization. The AdaGrad optimizer is used for the wider part. The combined model is illustrated in the figure (center).

Source	ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com