A dual graph convolutional neural network jointly considers the two essential assumptions of semi-supervised learning: (1) local consistency and (2) global consistency. Accordingly, two convolutional neural networks are devised to embed the local-consistency-based and global-consistency-based knowledge, respectively.

Description  and image from: [Dual Graph Convolutional Networks for Graph-Based Semi-Supervised Classification](https://persagen.com/files/misc/zhuang2018dual.pdf)

**LAMB** is a a layerwise adaptive large batch optimization technique. It provides a strategy for adapting the learning rate in large batch settings. LAMB uses [Adam](https://paperswithcode.com/method/adam) as the base algorithm and then forms an update as:

$$r\_{t} = \frac{m\_{t}}{\sqrt{v\_{t}} + \epsilon}$$
$$x\_{t+1}^{\left(i\right)} = x\_{t}^{\left(i\right)}  - \eta\_{t}\frac{\phi\left(|| x\_{t}^{\left(i\right)} ||\right)}{|| m\_{t}^{\left(i\right)} || }\left(r\_{t}^{\left(i\right)}+\lambda{x\_{t}^{\left(i\right)}}\right) $$

Unlike [LARS](https://paperswithcode.com/method/lars), the adaptivity of LAMB is two-fold: (i) per dimension normalization with respect to the square root of the second moment used in Adam and (ii) layerwise normalization obtained due to layerwise adaptivity.

LAMB

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

DualGCN

**Gaussian Gated Linear Network**, or **G-GLN**, is a multi-variate extension to the recently proposed [GLN](https://paperswithcode.com/method/gln) family of deep neural networks by reformulating the GLN neuron as a gated product of Gaussians. This Gaussian Gated Linear Network (G-GLN) formulation exploits the fact that exponential family densities are closed under multiplication, a property that has seen much use in [Gaussian Process](https://paperswithcode.com/method/gaussian-process) and related literature. Similar to the Bernoulli GLN, every neuron in the G-GLN directly predicts the target distribution.  

Precisely, a G-GLN is a feed-forward network of data-dependent distributions. Each neuron calculates the sufficient statistics $\left(\mu, \sigma\_{2}\right)$ for its associated PDF using its active weights, given those emitted by neurons in the preceding layer. It consists of consists of $L+1$ layers indexed by $i \in\{0, \ldots, L\}$ with $K\_{i}$ neurons in each layer. The weight space for a neuron in layer $i$ is denoted by $\mathcal{W}\_{i}$; the subscript is needed since the dimension of the weight space depends on $K_{i-1}$. Each neuron/distribution is indexed by its position in the network when laid out on a grid; for example, $f\_{i k}$ refers to the family of PDFs defined by the $k$ th neuron in the $i$ th layer. Similarly, $c\_{i k}$ refers to the context function associated with each neuron in layers $i \geq 1$, and $\mu\_{i k}$ and $\sigma\_{i k}^{2}$ (or $\Sigma\_{i k}$ in the multivariate case) referring to the sufficient statistics for each Gaussian PDF.

There are two types of input to neurons in the network. The first is the side information, which can be thought of as the input features, and is used to determine the weights used by each neuron via half-space gating. The second is the input to the neuron, which is the PDFs output by the previous layer, or in the case of layer 0, some provided base models. To apply a G-GLN in a supervised learning setting, we need to map the sequence of input-label pairs $\left(x\_{t}, y\_{t}\right)$ for $t=1,2, \ldots$ onto a sequence of (side information, base Gaussian PDFs, label) triplets $\left(z\_{t},\left$f\_{0 i}\right$\_{i}, y\_{t}\right)$. The side information $z\_{t}$ is set to the (potentially normalized) input features $x\_{t}$. The Gaussian PDFs for layer 0 will generally include the necessary base Gaussian PDFs to span the target range, and optionally some base prediction PDFs that capture domain-specific knowledge.

Year	2018
Data Source	CC BY-SA - https://paperswithcode.com