**Protagonist Antagonist Induced Regret Environment Design**, or **PAIRED**, is an adversarial method for approximate minimax regret to generate environments for reinforcement learning. It introduces an antagonist which is allied with the environment generating adversary. The primary agent we are trying to train is the protagonist. The environment adversary’s goal is to design environments in which the antagonist achieves high reward and the protagonist receives low reward. If the adversary generates unsolvable environments, the antagonist and protagonist would perform the same and the adversary would get a score of zero, but if the adversary finds environments the antagonist solves and the protagonist does not solve, the adversary achieves a positive score. Thus, the environment adversary is incentivized to create challenging but feasible environments, in which the antagonist can outperform the protagonist. Moreover, as the protagonist learns to solves the simple environments, the antagonist must generate more complex environments to make the protagonist fail, increasing the complexity of the generated tasks and leading to automatic curriculum generation.

Protagonist Antagonist Induced Regret Environment Design

Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

A **Window-based Discriminator** is a type of discriminator for generative adversarial networks. It is analogous to a [PatchGAN](https://paperswithcode.com/method/patchgan) but designed for audio. While a standard [GAN](https://paperswithcode.com/method/gan) discriminator learns to classify between distributions of entire audio sequences, window-based discriminator learns to classify between distribution of small audio chunks. Since the discriminator loss is computed over the overlapping windows where each window is very large (equal to the receptive field of the discriminator), the model learns to maintain coherence across patches.

Window-based Discriminator

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

**DeepLab** is a semantic segmentation architecture. First, the input image goes through the network with the use of dilated convolutions. Then the output from the network is bilinearly interpolated and goes through the fully connected [CRF](https://paperswithcode.com/method/crf) to fine tune the result we obtain the final predictions.

DeepLab

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Gated Graph Sequence Neural Networks (GGS-NNs) is a novel graph-based neural network model. GGS-NNs modifies Graph Neural Networks (Scarselli et al., 2009) to use gated recurrent units and modern optimization techniques and then extend to output sequences.

Source: [Li et al.](https://arxiv.org/pdf/1511.05493v4.pdf)

Image source: [Li et al.](https://arxiv.org/pdf/1511.05493v4.pdf)

GGS-NNs

Gated Graph Sequence Neural Networks

**Mobile Neural Network (MNN)** is a mobile inference engine tailored to mobile applications. The contributions of MNN include: (1) presenting a mechanism called pre-inference that manages to conduct runtime optimization; (2) delivering thorough kernel optimization on operators to achieve optimal computation performance; (3) introducing backend abstraction module which enables hybrid scheduling and keeps the engine lightweight.

Mobile Neural Network

MNN: A Universal and Efficient Inference Engine

**LightGCN** is a type of [graph convolutional neural network](https://paperswithcode.com/method/gcn) (GCN), including only the most essential component in GCN (neighborhood aggregation) for collaborative filtering. Specifically, LightGCN learns user and item embeddings by linearly propagating them on the user-item interaction graph, and uses the weighted sum of the embeddings learned at all layers as the final embedding.

LightGCN

LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation

**Mixed Attention Block** is an attention module used in the [ConvBERT](https://paperswithcode.com/method/convbert) architecture. It is a mixture of [self-attention](https://paperswithcode.com/method/scaled) and [span-based dynamic convolution](https://paperswithcode.com/method/span-based-dynamic-convolution) (highlighted in pink). They share the same Query but use different Key to generate the attention map and [convolution](https://paperswithcode.com/method/convolution) kernel respectively. The number of attention heads is reducing by directly projecting the input to a smaller embedding space to form a bottleneck structure for self-attention and span-based dynamic convolution. Dimensions of the input and output of some blocks are labeled on the left top corner to illustrate the overall framework, where $d$ is the embedding size of the input and $\gamma$ is the reduction ratio.

Mixed Attention Block

ConvBERT: Improving BERT with Span-based Dynamic Convolution

**GradDrop**, or **Gradient Sign Dropout**, is a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. It is applied as a layer in any standard network forward pass, usually on the final layer before the prediction head to save on compute overhead and maximize benefits during backpropagation. Below, we develop the GradDrop formalism. Throughout, o denotes elementwise multiplication after any necessary tiling operations (if any) are completed.
To implement GradDrop, we first define the Gradient Positive Sign Purity, $\mathcal{P}$, as

$$
\mathcal{P}=\frac{1}{2}\left(1+\frac{\sum\_{i} \nabla L_\{i}}{\sum\_{i}\left|\nabla L\_{i}\right|}\right)
$$

$\mathcal{P}$ is bounded by $[0,1] .$ For multiple gradient values $\nabla\_{a} L\_{i}$ at some scalar $a$, we see that $\mathcal{P}=0$ if $\nabla_{a} L\_{i}<0 $ $\forall i$, while $\mathcal{P}=1$ if $\nabla\_{a} L\_{i}>0$ $\forall i $. Thus, $\mathcal{P}$ is a measure of how many positive gradients are present at any given value. We then form a mask for each gradient $\mathcal{M}\_{i}$ as follows:

$$
\mathcal{M}\_{i}=\mathcal{I}[f(\mathcal{P})>U] \circ \mathcal{I}\left[\nabla L\_{i}>0\right]+\mathcal{I}[f(\mathcal{P})<U] \circ \mathcal{I}\left[\nabla L\_{i}<0\right]
$$

for $\mathcal{I}$ the standard indicator function and $f$ some monotonically increasing function (often just the identity) that maps $[0,1] \mapsto[0,1]$ and is odd around $(0.5,0.5)$. $U$ is a tensor composed of i.i.d $U(0,1)$ random variables. The $\mathcal{M}\_{i}$ is then used to produce a final gradient $\sum \mathcal{M}\_{i} \nabla L\_{i}$

GradDrop

Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout

**Performer** is a [Transformer](https://paperswithcode.com/methods/category/transformers) architectures which can estimate regular ([softmax](https://paperswithcode.com/method/softmax)) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. To approximate softmax attention-kernels, Performers use a Fast Attention Via positive Orthogonal Random features approach (FAVOR+), leveraging new methods for approximating softmax and Gaussian kernels.

Performer

Rethinking Attention with Performers

**EvoNorms** are a set of normalization-activation layers that go beyond existing design patterns. Normalization and activation are unified into a single computation graph, its structure is evolved starting from low-level primitives. EvoNorms consist of two series: B series and S series. The B series are batch-dependent and were discovered by our method without any constraint. The S series work on individual samples, and were discovered by rejecting any batch-dependent operations.

EvoNorms

Evolving Normalization-Activation Layers

**Inverse Square Root** is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where
$n$ is the current training iteration and $k$ is the number of warm-up steps. This sets a constant learning rate for the first $k$ steps, then exponentially decays the learning rate until pre-training is over.

Inverse Square Root Schedule

**Revision Network** is a style transfer module that aims to revise the rough stylized image via generating residual details image $r_{c s}$, while the final stylized image is generated by combining $r\_{c s}$ and rough stylized image $\bar{x}\_{c s}$. This procedure ensures that the distribution of global style pattern in $\bar{x}\_{c s}$ is properly kept. Meanwhile, learning to revise local style patterns with residual details image is easier for the Revision Network.

As shown in the Figure, the Revision Network is designed as a simple yet effective encoder-decoder architecture, with only one down-sampling and one up-sampling layer. Further, a [patch discriminator](https://paperswithcode.com/method/patchgan) is used to help Revision Network to capture fine patch textures under adversarial learning setting. The patch discriminator $D$ is defined following SinGAN, where $D$ owns 5 convolution layers and 32 hidden channels. A relatively shallow $D$ is chosen to (1) avoid overfitting since we only have one style image and (2) control the receptive field to ensure D can only capture local patterns.

Revision Network

Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer

**TABBIE** is a pretraining objective (*corrupt cell detection*) that learns exclusively from tabular data. Unlike other approaches, TABBIE provides embeddings of all table substructures (cells, rows, and columns). TABBIE can be seen as a table embedding model trained to detect corrupted cells, inspired by the [ELECTRA](https://www.paperswithcode.com/method/electra) objective function.

TABBIE

TABBIE: Pretrained Representations of Tabular Data

A regularization criterion that, differently from [dropout](https://paperswithcode.com/method/dropout) and its variants, is deterministic rather than random. It grounds on the empirical evidence that feature descriptors with larger L2-norm and highly-active nodes are strongly correlated to confident class predictions. Thus, the criterion guides towards dropping a percentage of the most active nodes of the descriptors, proportionally to the estimated class probability

Attack on image classifiers by a image solarization through greedy random search.

RandSol

Don't Look into the Sun: Adversarial Solarization Attacks on Image Classifiers

**Parametric UMAP** is a non-parametric graph-based dimensionality reduction algorithm that extends the second step of [UMAP](https://www.paperswithcode.com/method/umap) to a parametric optimization over neural network weights, learning a parametric relationship between data and embedding.

Parametric UMAP

Parametric UMAP embeddings for representation and semi-supervised learning

**Fully Convolutional Networks**, or **FCNs**, are an architecture used mainly for semantic segmentation. They employ solely locally connected layers, such as [convolution](https://paperswithcode.com/method/convolution), pooling and upsampling. Avoiding the use of dense layers means less parameters (making the networks faster to train). It also means an FCN can work for variable image sizes given all connections are local.

The network consists of a downsampling path, used to extract and interpret the context, and an upsampling path, which allows for localization. 

FCNs also employ skip connections to recover the fine-grained spatial information lost in the downsampling path.

Fully Convolutional Networks for Semantic Segmentation

The **Pointer Sentinel-LSTM mixture model** is a type of recurrent neural network that combines the advantages of standard [softmax](https://paperswithcode.com/method/softmax) classifiers with those of a pointer component for effective and efficient language modeling. Rather than relying on the RNN hidden state to decide when to use the pointer, the model allows the pointer component itself to decide when to use the softmax vocabulary through a sentinel.

AI Glossary