An **EESP Unit**, or  Extremely Efficient Spatial Pyramid of Depth-wise Dilated Separable Convolutions, is an image model block designed for edge devices. It was proposed as part of the [ESPNetv2](https://paperswithcode.com/method/espnetv2) CNN architecture. 

This building block is based on a reduce-split-transform-merge strategy. The EESP unit first projects the high-dimensional input feature maps into low-dimensional space using groupwise pointwise convolutions and then learns the representations in parallel using depthwise dilated separable convolutions with different dilation rates. Different dilation rates in each branch allow the EESP unit to learn the representations from a large effective receptive field. To remove the gridding artifacts caused by dilated convolutions, the EESP fuses the feature maps using [hierarchical feature fusion](https://paperswithcode.com/method/hierarchical-feature-fusion) (HFF).

Contextual Graph Markov Model (CGMM) is an approach combining ideas from generative models and neural networks for the processing of graph data. It founds on a constructive methodology to build a deep architecture comprising layers of probabilistic models that learn to encode the structured information in an incremental fashion. Context is diffused in an efficient and scalable way across the graph vertexes and edges. The resulting graph encoding is used in combination with discriminative models to address structure classification benchmarks.

Description and image from: [Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing](https://arxiv.org/pdf/1805.10636.pdf)

CGMM

Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing

EESP

ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

**Contrastive Predictive Coding (CPC)** learns self-supervised representations by predicting the future in latent space by using powerful autoregressive models. The model uses a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful
to predict future samples.

First, a non-linear encoder $g\_{enc}$ maps the input sequence of observations $x\_{t}$ to a sequence of latent representations $z\_{t} = g\_{enc}\left(x\_{t}\right)$, potentially with a lower temporal resolution. Next, an autoregressive model $g\_{ar}$ summarizes all $z\leq{t}$ in the latent space and produces a context latent representation $c\_{t} = g\_{ar}\left(z\leq{t}\right)$.

A density ratio is modelled which preserves the mutual information between $x\_{t+k}$ and $c\_{t}$ as follows:

$$ f\_{k}\left(x\_{t+k}, c\_{t}\right) \propto \frac{p\left(x\_{t+k}|c\_{t}\right)}{p\left(x\_{t+k}\right)} $$

where $\propto$ stands for ’proportional to’ (i.e. up to a multiplicative constant). Note that the density ratio $f$ can be unnormalized (does not have to integrate to 1). The authors use a simple log-bilinear model:

$$ f\_{k}\left(x\_{t+k}, c\_{t}\right) = \exp\left(z^{T}\_{t+k}W\_{k}c\_{t}\right) $$

Any type of autoencoder and autoregressive can be used. An example the authors opt for is strided convolutional layers with residual blocks and GRUs.

The autoencoder and autoregressive models are trained to minimize an [InfoNCE](https://paperswithcode.com/method/infonce) loss (see components).

Source	ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com