**Xception** is a convolutional neural network architecture that relies solely on [depthwise separable convolution](https://paperswithcode.com/method/depthwise-separable-convolution) layers.

Xception

Xception: Deep Learning With Depthwise Separable Convolutions

A **Spatial Attention Module** is a module for spatial attention in convolutional neural networks. It generates a spatial attention map by utilizing the inter-spatial relationship of features. Different from the [channel attention](https://paperswithcode.com/method/channel-attention-module), the spatial attention focuses on where is an informative part, which is complementary to the channel attention. To compute the spatial attention, we first apply average-pooling and max-pooling operations along the channel axis and concatenate them to generate an efficient feature descriptor. On the concatenated feature descriptor, we apply a [convolution](https://paperswithcode.com/method/convolution) layer to generate a spatial attention map $\textbf{M}\_{s}\left(F\right) \in \mathcal{R}^{H×W}$ which encodes where to emphasize or suppress. 

We aggregate channel information of a feature map by using two pooling operations, generating two 2D maps: $\mathbf{F}^{s}\_{avg} \in \mathbb{R}^{1\times{H}\times{W}}$ and $\mathbf{F}^{s}\_{max} \in \mathbb{R}^{1\times{H}\times{W}}$. Each denotes average-pooled features and max-pooled features across the channel. Those are then concatenated and convolved by a standard convolution layer, producing the 2D spatial attention map. In short, the spatial attention is computed as:

$$ \textbf{M}\_{s}\left(F\right) = \sigma\left(f^{7x7}\left(\left[\text{AvgPool}\left(F\right);\text{MaxPool}\left(F\right)\right]\right)\right) $$

$$ \textbf{M}\_{s}\left(F\right) = \sigma\left(f^{7x7}\left(\left[\mathbf{F}^{s}\_{avg};\mathbf{F}^{s}\_{max} \right]\right)\right) $$

where $\sigma$ denotes the sigmoid function and $f^{7×7}$ represents a convolution operation with the filter size of 7 × 7.

Spatial Attention Module

CBAM: Convolutional Block Attention Module

A closed-loop evaluation procedure is first used in a simulator to identify training data samples that are important for practical driving performance and then we these samples to help debias the policy network.

CW-ERM

CW-ERM: Improving Autonomous Driving Planning with Closed-loop Weighted Empirical Risk Minimization

Fast-OCR is a new lightweight detection network that incorporates features from existing models focused on the speed/accuracy trade-off, such as [YOLOv2](https://paperswithcode.com/method/yolov2), [CR-NET](https://paperswithcode.com/method/cr-net), and Fast-[YOLOv4](https://paperswithcode.com/method/yolov4).

Fast-OCR

Towards Image-based Automatic Meter Reading in Unconstrained Scenarios: A Robust and Efficient Approach

**NoisyNet-DQN** is a modification of a [DQN](https://paperswithcode.com/method/dqn) that utilises noisy linear layers for exploration instead of $\epsilon$-greedy exploration as in the original DQN formulation.

NoisyNet-DQN

Noisy Networks for Exploration

A **Feature Pyramid Network**, or **FPN**, is a feature extractor that takes a single-scale image of an arbitrary size as input, and outputs proportionally sized feature maps at multiple levels, in a fully convolutional fashion. This process is independent of the backbone convolutional architectures. It therefore acts as a generic solution for building feature pyramids inside deep convolutional networks to be used in tasks like object detection.

The construction of the pyramid involves a bottom-up pathway and a top-down pathway.

The bottom-up pathway is the feedforward computation of the backbone ConvNet, which computes a feature hierarchy consisting of feature maps at several scales with a scaling step of 2. For the feature
pyramid, one pyramid level is defined for each stage. The output of the last layer of each stage is used as a reference set of feature maps. For [ResNets](https://paperswithcode.com/method/resnet) we use the feature activations output by each stage’s last [residual block](https://paperswithcode.com/method/residual-block). 

The top-down pathway hallucinates higher resolution features by upsampling spatially coarser, but semantically stronger, feature maps from higher pyramid levels. These features are then enhanced with features from the bottom-up pathway via lateral connections. Each lateral connection merges feature maps of the same spatial size from the bottom-up pathway and the top-down pathway. The bottom-up feature map is of lower-level semantics, but its activations are more accurately localized as it was subsampled fewer times.

Feature Pyramid Networks for Object Detection

An **Autoencoder** is a bottleneck architecture that turns a high-dimensional input into a latent low-dimensional code (encoder), and then performs a reconstruction of the input with this latent code (the decoder).

Image: [Michael Massi](https://en.wikipedia.org/wiki/Autoencoder#/media/File:Autoencoder_schema.png)

AutoEncoder

Reducing the Dimensionality of Data with Neural Networks

**One-Shot Aggregation** is an image model block that is an alternative to [Dense Blocks](https://paperswithcode.com/method/dense-block), by aggregating intermediate features. It is proposed as part of the [VoVNet](https://paperswithcode.com/method/vovnet) architecture. Each [convolution](https://paperswithcode.com/method/convolution) layer is connected by two-way connection. One way is connected to the subsequent layer to produce the feature with a larger receptive field while the other way is aggregated only once into the final output feature map. The difference with [DenseNet](https://paperswithcode.com/method/densenet) is that the output of each layer is not routed to all subsequent intermediate layers which makes the input size of intermediate layers constant.

One-Shot Aggregation

An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection

An **Invertible Rescaling Network (IRN)** is a network for image rescaling.  According to the Nyquist-Shannon sampling theorem, high-frequency contents are lost during downscaling. Ideally, we hope to keep all lost information to perfectly recover the original HR image, but storing or transferring the high-frequency information is unacceptable. In order to well address this challenge, the Invertible Rescaling Net (IRN) captures some knowledge on the lost information in the form of its distribution and embeds it into model’s parameters to mitigate the ill-posedness. Given an HR image $x$, IRN not only downscales it into a LR image y, but also embeds the case-specific high-frequency content into an auxiliary case-agnostic latent variable $z$, whose marginal distribution
obeys a fixed pre-specified distribution (e.g., isotropic Gaussian). Based on this model,
we use a randomly drawn sample of $z$ from the pre-specified distribution for the inverse upscaling procedure, which holds the most information that one could have in upscaling.

Invertible Image Rescaling

The methon to overcome catastrophic forgetting in neural network while continual learning

Overcoming catastrophic forgetting in neural networks

**Self-Cure Network**, or **SCN**, is a method for suppressing uncertainties for large-scale facial expression recognition, prventing deep networks from overfitting uncertain facial images. Specifically, SCN suppresses the uncertainty from two different aspects: 1) a self-attention mechanism over mini-batch to weight each training sample with a ranking regularization, and 2) a careful relabeling mechanism to modify the labels of these samples in the lowest-ranked group.

Suppressing Uncertainties for Large-Scale Facial Expression Recognition

A **Switch FFN** is a sparse layer that operates independently on tokens within an input sequence. It is shown in the blue block in the figure. We diagram two tokens ($x\_{1}$ = “More” and $x\_{2}$ = “Parameters” below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).

Switch FFN

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

In recent years, there has been a growing interest on deep learning-based pansharpening.
Research has mainly focused on architectures.
However, lacking a ground truth, model training is also a major issue.
A popular approach is to train networks in a reduced resolution domain, using the original data as ground truths.
The trained networks are then used on full resolution data, relying on an implicit scale invariance hypothesis.
Results are generally good at reduced resolution, but more questionable at full resolution.

Here, we propose a full-resolution training framework for deep learning-based pansharpening.
Training takes place in the high resolution domain, relying only on the original data, with no loss of information.
To ensure spectral and spatial fidelity, suitable losses are defined,
which force the pansharpened output to be consistent with the available panchromatic and multispectral input.
Experiments carried out on WorldView-3, WorldView-2, and GeoEye-1 images show that methods trained with the proposed framework
guarantee an excellent performance in terms of both full-resolution numerical indexes and visual quality.
The framework is fully general, and can be used to train and fine-tune any deep learning-based pansharpening network.

Z-PNN

Pansharpening by convolutional neural networks in the full resolution framework

**Inception-ResNet-v2 Reduction-B** is an image model block used in the [Inception-ResNet-v2](https://paperswithcode.com/method/inception-resnet-v2) architecture.

Inception-ResNet-v2 Reduction-B

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

**Residual Networks**, or **ResNets**, learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. Instead of hoping each few stacked layers directly fit a desired underlying mapping, residual nets let these layers fit a residual mapping. They stack [residual blocks](https://paperswithcode.com/method/residual-block) ontop of each other to form network: e.g. a ResNet-50 has fifty layers using these blocks. 

Formally, denoting the desired underlying mapping as $\mathcal{H}(x)$, we let the stacked nonlinear layers fit another mapping of $\mathcal{F}(x):=\mathcal{H}(x)-x$. The original mapping is recast into $\mathcal{F}(x)+x$.

There is empirical evidence that these types of network are easier to optimize, and can gain accuracy from considerably increased depth.

ResNet

Deep Residual Learning for Image Recognition

**RMSProp** is an unpublished adaptive learning rate optimizer [proposed by Geoff Hinton](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:

$$E\left[g^{2}\right]\_{t} = \gamma E\left[g^{2}\right]\_{t-1} + \left(1 - \gamma\right) g^{2}\_{t}$$

$$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]\_{t} + \epsilon}}g\_{t}$$

Hinton suggests $\gamma=0.9$, with a good default for $\eta$ as $0.001$.

Image: [Alec Radford](https://twitter.com/alecrad)

RMSProp

**Stochastic Weight Averaging** is an optimization procedure that averages multiple points along the trajectory of [SGD](https://paperswithcode.com/method/sgd), with a cyclical or constant learning rate. On the one hand it averages weights, but it also has the property that, with a cyclical or constant learning rate, SGD proposals are approximately sampling from the loss surface of the network, leading to stochastic weights and helping to discover broader optima.

Stochastic Weight Averaging

Averaging Weights Leads to Wider Optima and Better Generalization

**Pythia** is a suite of decoder-only autoregressive language models all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. The model architecture and hyperparameters largely follow GPT-3, with a few notable deviations based on recent advances in best practices for large scale language modeling.

AI Glossary

Xception

Spatial Attention Module

Closed-loop Weighted Empirical Risk Minimization

Fast-OCR

NoisyNet-DQN

Feature Pyramid Network

AutoEncoder

One-Shot Aggregation

Invertible Rescaling Network

Elastic Weight Consolidation

Self-Cure Network

Switch FFN

Pansharpening by convolutional neural networks in the full resolution framework

Inception-ResNet-v2 Reduction-B

Residual Network

RMSProp

Stochastic Weight Averaging

Pythia