**LightConv** is a type of [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) for sequential modelling which shares certain output channels and whose weights are normalized across the temporal dimension using a [softmax](https://paperswithcode.com/method/softmax). Compared to self-attention, LightConv has a fixed context window and it determines the importance of context elements with a set of weights that do not change over time steps. LightConv computes the following for the $i$-th element in the sequence and output channel $c$:

$$ \text{LightConv}\left(X, W\_{\text{ceil}\left(\frac{cH}{d}\right),:}, i, c\right) = \text{DepthwiseConv}\left(X,\text{softmax}\left(W\_{\text{ceil}\left(\frac{cH}{d}\right),:}\right), i, c\right) $$

This method introduces the aggregated dense block within the squeeze excitation block to enhance representation. The squeeze method compresses the input flow and sends it to excitation with dense layers to regain its shape. The paper introduces multiple dense layers stacked side by side, similar to ResNeXt. This learns global representations from the condensed information which enhances the representational power of the network.

SAENet

Squeeze aggregated excitation network

LightConv

Pay Less Attention with Lightweight and Dynamic Convolutions

**Graph Echo State Network** (**GraphESN**) model is a generalization of the Echo State Network (ESN) approach to graph domains. GraphESNs allow for an efficient approach to Recursive Neural Networks (RecNNs) modeling extended to deal with cyclic/acyclic, directed/undirected, labeled graphs. The recurrent reservoir of the network computes a fixed contractive encoding function over graphs and is left untrained after initialization, while a feed-forward readout implements an adaptive linear output function. Contractivity of the state transition function implies a Markovian characterization of state dynamics and stability of the state computation in presence of cycles. Due to the use of fixed (untrained) encoding, the model represents both an extremely efficient version and a baseline for the performance of recursive models with trained connections.

Description from: [Graph Echo State Networks](https://ieeexplore.ieee.org/document/5596796)

Source	Pay Less Attention with Lightweight and Dynamic Convolutions
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com