Inspired by the success of ResNet,
Wang et al. proposed
the very deep convolutional residual attention network (RAN) by 
combining an attention mechanism with residual connections. 

Each attention module stacked in a residual attention network 
can be divided into a mask branch and a trunk branch. 
The trunk branch processes features,
and can be implemented by any state-of-the-art structure
including a pre-activation residual unit and an inception block.
The mask branch uses a bottom-up top-down structure
to learn a mask of the same size that 
softly weights output features from the trunk branch. 
A sigmoid layer normalizes the output to $[0,1]$ after two $1\times 1$ convolution layers. Overall the residual attention mechanism can be written as

\begin{align}
s &= \sigma(Conv_{2}^{1\times 1}(Conv_{1}^{1\times 1}( h_\text{up}(h_\text{down}(X))))) 
\end{align}

\begin{align}
X_{out} &= s f(X) + f(X)
\end{align}
where $h_\text{up}$ is a bottom-up structure, 
using max-pooling several times after residual units
to increase the receptive field, while
$h_\text{down}$ is the top-down part using 
linear interpolation to keep the output size the 
same as the input feature map. 
There are also skip-connections between the two parts,
which are omitted from the formulation.
$f$ represents the trunk branch
which can be any state-of-the-art structure.

Inside each attention module, a
bottom-up top-down feedforward structure models
both spatial and cross-channel dependencies, 
 leading to a consistent performance improvement. 
Residual attention can be incorporated into
any deep network structure in an end-to-end training fashion.
However, the proposed bottom-up top-down structure fails to leverage global spatial information.  
Furthermore, directly predicting a 3D attention map  has high computational cost.

Residual Attention Network for Image Classification

Fast-BAT is a new method for accelerated adversarial training.

Fast_BAT

Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization

**Logistic Regression**, despite its name, is a linear model for classification rather than regression. Logistic regression is also known in the literature as logit regression, maximum-entropy classification (MaxEnt) or the log-linear classifier. In this model, the probabilities describing the possible outcomes of a single trial are modeled using a logistic function.

Source: [scikit-learn](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression)

Image: [Michaelg2015](https://commons.wikimedia.org/wiki/User:Michaelg2015)

Logistic Regression

**Rotary Position Embedding**, or **RoPE**, is a type of position embedding which encodes absolute positional information with rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and capability of equipping the linear self-attention with relative position encoding.

Rotary Embeddings

RoFormer: Enhanced Transformer with Rotary Position Embedding

**FixUp Initialization**, or **Fixed-Update Initialization**, is an initialization method that rescales the standard initialization of [residual branches](https://paperswithcode.com/method/residual-block) by adjusting for the network architecture. Fixup aims to enables training very deep [residual networks](https://paperswithcode.com/method/resnet) stably at a maximal learning rate without [normalization](https://paperswithcode.com/methods/category/normalization).

The steps are as follows:

1. Initialize the classification layer and the last layer of each residual branch to 0.

2. Initialize every other layer using a standard method, e.g. [Kaiming Initialization](https://paperswithcode.com/method/he-initialization), and scale only the weight layers inside residual branches by $L^{\frac{1}{2m-2}}$.

3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each [convolution](https://paperswithcode.com/method/convolution), [linear](https://paperswithcode.com/method/linear-layer), and element-wise activation layer.

Fixup Initialization

Fixup Initialization: Residual Learning Without Normalization

**TrIVD-GAN**, or **Transformation-based & TrIple Video Discriminator GAN**, is a type of generative adversarial network for video generation that builds upon [DVD-GAN](https://paperswithcode.com/method/dvd-gan). Improvements include a novel transformation-based recurrent unit (the TSRU) that makes the generator more expressive, and an improved discriminator architecture. 

In contrast with DVD-[GAN](https://paperswithcode.com/method/gan), TrIVD-GAN has an alternative split for the roles of the discriminators, with $\mathcal{D}\_{S}$ judging per-frame global structure, while $\mathcal{D}\_{T}$ critiques local spatiotemporal structure. This is achieved by downsampling the $k$ randomly sampled frames fed to $\mathcal{D}\_{S}$ by a factor $s$, and cropping $T \times H/s \times W/s$ clips inside the high resolution video fed to $\mathcal{D}\_{T}$, where $T, H, W, C$ correspond to time, height, width and channel dimension of the input. This further reduces the number of pixels to process per video,
from $k \times H \times W + T \times H/s \times W/s$ to $\left(k + T\right) \times H/s \times W/s$.

TrIVD-GAN

Transformation-based Adversarial Video Prediction on Large-Scale Data

An unsupervised approach for identifying Hierarchical Information Threads by analysing the network of related articles in a collection. In particular, HINT leverages article timestamps and the 5W1H questions to identify related articles about an event or discussion. HINT then constructs a network representation of the articles,
and identify threads as strongly connected hierarchical network communities.

HINT

Effective Hierarchical Information Threading Using Network Community Detection

**PSANet** is a semantic segmentation architecture that utilizes a [Point-wise Spatial Attention](https://paperswithcode.com/method/point-wise-spatial-attention) (PSA) module to aggregate long-range contextual information in a flexible and adaptive manner. Each position in the feature map is connected with all other ones through self-adaptively predicted attention maps, thus harvesting various information nearby and far away. Furthermore, the authors design the bi-directional information propagation path for a comprehensive understanding of complex scenes. Each position collects information from all others to help the prediction of itself and vice versa, the information at each position can be distributed globally, assisting the prediction of all other positions. Finally, the bi-directionally aggregated contextual information is fused with local features to form the final representation of complex scenes.

The authors use [ResNet](https://paperswithcode.com/method/resnet) as an [FCN](https://paperswithcode.com/method/fcn) backbone for PSANet, as the Figure to the right illustrates. The proposed PSA module is then used to aggregate long-range contextual information from the local representation. It follows stage-5 in ResNet, which is the final stage of the FCN backbone. Features in stage-5 are semantically stronger. Aggregating them together leads to a more comprehensive representation of long-range context. Moreover, the spatial size of the feature map at stage-5 is smaller and can reduce computation overhead and memory consumption. An auxiliary loss branch is applied apart from the main loss.

PSANet

PSANet: Point-wise Spatial Attention Network for Scene Parsing

**SkipInit** is a method that aims to allow [normalization](https://paperswithcode.com/methods/category/normalization)-free training of neural networks by downscaling [residual branches](https://paperswithcode.com/method/residual-block) at initialization.  This is achieved by including a learnable scalar multiplier at the end of each residual branch, initialized to $\alpha$.

The method is motivated by theoretical findings that [batch normalization](https://paperswithcode.com/method/batch-normalization) downscales the hidden activations on the residual branch by a factor on the order of the square root of the network depth (at initialization). Therefore, as the depth of a residual network is increased, the residual blocks are increasingly dominated by the [skip connection](https://paperswithcode.com/method/residual-connection), which drives the functions computed by residual blocks closer to the identity, preserving signal propagation and ensuring well-behaved gradients. This leads to the proposed method which can achieve this property through an [initialization](https://paperswithcode.com/methods/category/initialization) strategy rather than a [normalization](https://paperswithcode.com/methods/category/normalization) strategy.

SkipInit

Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

VLG-Net leverages recent advantages in Graph Neural Networks (GCNs) and leverages a novel multi-modality graph-based fusion method for the task of natural language video grounding.

VLG-Net

VLG-Net: Video-Language Graph Matching Network for Video Grounding

**Adversarial Color Enhancement** is an approach to generating unrestricted adversarial images by optimizing a color filter via gradient descent.

Adversarial Color Enhancement

Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

**Group Decreasing Network**, or **GroupDNet**, is a type of convolutional neural network for multi-modal image synthesis. GroupDNet contains one encoder and one decoder. Inspired by the idea of [VAE](https://paperswithcode.com/method/vae) and SPADE, the encoder $E$ produces a
latent code $Z$ that is supposed to follow a Gaussian distribution $\mathcal{N}(0,1)$ during training. While testing, the encoder $E$ is discarded. A randomly sampled code from the Gaussian distribution substitutes for $Z$. To fulfill this, the re-parameterization trick is used to enable a differentiable loss function during training. Specifically, the encoder predicts a mean vector and a variance vector through two fully connected layers to represent the encoded distribution. The gap between the encoded distribution and Gaussian distribution can be minimized by imposing a KL-divergence loss.

GroupDNet

Semantically Multi-modal Image Synthesis

The **alternating direction method of multipliers** (**ADMM**) is an algorithm that solves convex optimization problems by breaking them into smaller pieces, each of which are then easier to handle. It takes the form of a decomposition-coordination procedure, in which the solutions to small
local subproblems are coordinated to find a solution to a large global problem. ADMM can be viewed as an attempt to blend the benefits of dual decomposition and augmented Lagrangian methods for constrained optimization. It turns out to be equivalent or closely related to many other algorithms
as well, such as Douglas-Rachford splitting from numerical analysis, Spingarn’s method of partial inverses, Dykstra’s alternating projections method, Bregman iterative algorithms for l1 problems in signal processing, proximal methods, and many others.

Text Source: [https://stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf](https://stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf)

Image Source: [here](https://www.slideshare.net/derekcypang/alternating-direction)

ADMM

Galactica is a language model which uses a Transformer architecture in a decoder-only setup with the following modifications:

- It uses GeLU activations on all model sizes
- It uses a 2048 length context window for all model sizes
- It does not use biases in any of the dense kernels or layer norms
- It uses learned positional embeddings for the model
- A vocabulary of 50k tokens was constructed using BPE. The vocabulary was generated from a randomly selected 2% subset of the training data

Galactica

Galactica: A Large Language Model for Science

**Two-Way Dense Layer** is an image model block used in the [PeleeNet](https://paperswithcode.com/method/peleenet) architectures. Motivated by [GoogLeNet](https://paperswithcode.com/method/googlenet), the 2-way dense layer is used to get different scales of receptive fields. One way of the layer uses a 3x3 kernel size. The other way of the layer uses two stacked 3x3 [convolution](https://paperswithcode.com/method/convolution) to learn visual patterns for large objects.

Two-Way Dense Layer

Pelee: A Real-Time Object Detection System on Mobile Devices

**Ape-X DQN** is a variant of a [DQN](https://paperswithcode.com/method/dqn) with some components of [Rainbow-DQN](https://paperswithcode.com/method/rainbow-dqn) that utilizes distributed [prioritized experience replay](https://paperswithcode.com/method/prioritized-experience-replay) through the [Ape-X](https://paperswithcode.com/method/ape-x) architecture.

Ape-X DQN

Distributed Prioritized Experience Replay

Deep Neural Networks (DNNs) are typically trained by backpropagation in a batch learning setting, which requires the entire training data to be made available prior to the learning task. This is not scalable for many real-world scenarios where new data arrives sequentially in a stream form. We aim to address an open challenge of "Online Deep Learning" (ODL) for learning DNNs on the fly in an online setting. Unlike traditional online learning that often optimizes some convex objective function with respect to a shallow model (e.g., a linear/kernel-based hypothesis), ODL is significantly more challenging since the optimization of the DNN objective function is non-convex, and regular backpropagation does not work well in practice, especially for online learning settings.

Online Deep Learning: Learning Deep Neural Networks on the Fly

GLIDE is a generative model based on text-guided diffusion models for more photorealistic image generation. Guided diffusion is applied to text-conditional image synthesis and the model is able to handle free-form prompts. The diffusion model uses a text encoder to condition on natural language descriptions. The model is provided with editing capabilities in addition to zero-shot generation, allowing for iterative improvement of model samples to match more complex prompts. The model is fine-tuned to perform image inpainting.

AI Glossary

Residual Attention Network

Fast Bi-level Adversarial Training

Logistic Regression

Rotary Position Embedding

Fixup Initialization

TrIVD-GAN

Hierarchical Information Threading

PSANet

SkipInit

Video Language Graph Matching Network

Adversarial Color Enhancement

Group Decreasing Network

Alternating Direction Method of Multipliers

Galactica

Two-Way Dense Layer

Ape-X DQN

online deep learning

Guided Language to Image Diffusion for Generation and Editing