**MobileNetV2** is a convolutional neural network architecture that seeks to perform well on mobile devices. It is based on an inverted residual structure where the residual connections are between the bottleneck layers.  The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. As a whole, the architecture of MobileNetV2 contains the initial fully [convolution](https://paperswithcode.com/method/convolution) layer with 32 filters, followed by 19 residual bottleneck layers.

MobileNetV2

MobileNetV2: Inverted Residuals and Linear Bottlenecks

A **Dual Path Network (DPN)** is a convolutional neural network which presents a new topology of connection paths internally. The intuition is that [ResNets](https://paperswithcode.com/method/resnet) enables feature re-usage while [DenseNet](https://paperswithcode.com/method/densenet) enables new feature exploration, and both are important for learning good representations. To enjoy the benefits from both path topologies, Dual Path Networks share common features while maintaining the flexibility to explore new features through dual path architectures. 

We formulate such a dual path architecture as follows:

$$x^{k} = \sum\limits\_{t=1}^{k-1} f\_t^{k}(h^t) \text{,}  $$

$$
y^{k} = \sum\limits\_{t=1}^{k-1} v\_t(h^t) = y^{k-1} + \phi^{k-1}(y^{k-1}) \text{,} \\\\
$$

$$
r^{k} = x^{k} + y^{k} \text{,} \\\\
$$

$$
h^k = g^k \left( r^{k} \right) \text{,}
$$

where $x^{k}$ and $y^{k}$ denote the extracted information at $k$-th step from individual path, $v_t(\cdot)$ is a feature learning function as $f_t^k(\cdot)$. The first equation refers to the densely connected path that enables exploring new features. The second equation refers to the residual path that enables common features re-usage. The third equation defines the dual path that integrates them and feeds them to the last transformation function in the last equation.

Dual Path Networks

**SHA-RNN**, or **Single Headed Attention RNN**, is a recurrent neural network, and language model when combined with an embedding input and [softmax](https://paperswithcode.com/method/softmax) classifier, based on a core [LSTM](https://paperswithcode.com/method/lstm) component and a [single-headed attention](https://paperswithcode.com/method/single-headed-attention) module. Other design choices include a Boom feedforward layer and the use of [layer normalization](https://paperswithcode.com/method/layer-normalization). The guiding principles of the author were to ensure simplicity in the architecture and to keep computational costs bounded (the model was originally trained with a single GPU).

SHA-RNN

Single Headed Attention RNN: Stop Thinking With Your Head

**Social-STGCNN** is a method for human trajectory prediction. Pedestrian trajectories are not only influenced by the pedestrian itself but also by interaction with surrounding objects.

Social-STGCNN

Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction

**ACER**, or **Actor Critic with Experience Replay**, is an actor-critic deep reinforcement learning agent with [experience replay](https://paperswithcode.com/method/experience-replay). It can be seen as an off-policy extension of [A3C](https://paperswithcode.com/method/a3c), where the off-policy estimator is made feasible by:

- Using [Retrace](https://paperswithcode.com/method/retrace) Q-value estimation.
- Using truncated importance sampling with bias correction.
- Using a trust region policy optimization method.
- Using a [stochastic dueling network](https://paperswithcode.com/method/stochastic-dueling-network) architecture.

ACER

Sample Efficient Actor-Critic with Experience Replay

NICE-SLAM, a dense RGB-D SLAM system that combines neural implicit decoders with hierarchical grid-based representations, which can be applied to large-scale scenes.

Neural implicit representations have recently shown encouraging results in various domains, including promising progress in simultaneous localization and mapping (SLAM). Nevertheless, existing methods produce over-smoothed scene reconstructions and have difficulty scaling up to large scenes. These limitations are mainly due to their simple fully-connected network architecture that does not incorporate local information in the observations. In this paper, we present NICE-SLAM, a dense SLAM system that incorporates multi-level local information by introducing a hierarchical scene representation. Optimizing this representation with pre-trained geometric priors enables detailed reconstruction on large indoor scenes. Compared to recent neural implicit SLAM systems, our approach is more scalable, efficient, and robust. Experiments on five challenging datasets demonstrate competitive results of NICE-SLAM in both mapping and tracking quality.

NICE-SLAM

NICE-SLAM: Neural Implicit Scalable Encoding for SLAM

**$L_{1}$ Regularization** is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a penalty on the $L\_{1}$ Norm of the weights:

$$L\_{new}\left(w\right) = L\_{original}\left(w\right) + \lambda{||w||}\_{1}$$

where $\lambda$ is a value determining the strength of the penalty. In contrast to [weight decay](https://paperswithcode.com/method/weight-decay), $L_{1}$ regularization promotes sparsity; i.e. some parameters have an optimal value of zero.

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Regularization_(mathematics)#/media/File:Sparsityl1.png)

L1 Regularization

**modReLU** is an activation that is a modification of a [ReLU](https://paperswithcode.com/method/relu). It is a pointwise nonlinearity, $\sigma\_{modReLU}\left(z\right) : C \rightarrow C$, which affects only the absolute value of a complex number, defined as:

$$ \sigma\_{modReLU}\left(z\right) = \left(|z| + b\right)\frac{z}{|z|} \text{ if } |z| + b \geq 0 $$
$$ \sigma\_{modReLU}\left(z\right) = 0 \text{ if } |z| + b \leq 0 $$

where $b \in \mathbb{R}$ is a bias parameter of the nonlinearity. For a $n\_{h}$ dimensional hidden space we learn $n\_{h}$ nonlinearity bias parameters, one per dimension.

modReLU

Unitary Evolution Recurrent Neural Networks

**Mode Normalization** extends normalization to more than a single mean and variance, allowing for detection of modes of data on-the-fly, jointly normalizing samples that share common features. It first assigns samples in a mini-batch to different modes via a gating network, and then normalizes each sample with estimators for its corresponding mode.

Mode Normalization

An **Recursive Feature Pyramid (RFP)** builds on top of the Feature Pyramid Networks ([FPN](https://paperswithcode.com/method/fpn)) by incorporating extra feedback connections from the FPN layers into the bottom-up backbone layers. Unrolling the recursive structure to a sequential implementation, we obtain a backbone for object detector that looks at the images twice or more. Similar to the cascaded detector heads in [Cascade R-CNN](https://paperswithcode.com/method/cascade-r-cnn) trained with more selective examples, an RFP recursively enhances FPN to generate increasingly powerful representations. Resembling Deeply-Supervised Nets, the feedback connections bring the features that directly receive gradients from the detector heads back to the low levels of the bottom-up backbone to speed up training and boost performance.

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

VERtex Similarity Embeddings (VERSE) is a simple, versatile, and memory-efficient method that derives graph embeddings explicitly calibrated to preserve the distributions of a selected vertex-to-vertex similarity measure. VERSE learns such embeddings by training a single-layer neural network.

Source: [Tsitsulin et al.](https://arxiv.org/pdf/1803.04742v1.pdf)

Image source: [Tsitsulin et al.](https://arxiv.org/pdf/1803.04742v1.pdf)

VERSE

VERSE: Versatile Graph Embeddings from Similarity Measures

**ENIGMA** is an evaluation framework for dialog systems based on Pearson and Spearman's rank correlations between the estimated rewards and the true rewards.  ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More importantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experience data (see details in Section 2), which significantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors.

ENIGMA

Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach

**Voxel RoI Pooling** is a RoI feature extractor extracts RoI features directly from voxel features for further refinement. It starts by dividing a region proposal into $G \times G \times G$ regular sub-voxels. The center point is taken as the grid point of the corresponding sub-voxel. Since $3 D$ feature volumes are extremely sparse (non-empty voxels account for $<3 \%$ spaces), we cannot directly utilize max pooling over features of each sub-voxel. Instead, features are integrated from neighboring voxels into the grid points for feature extraction. Specifically, given a grid point $g\_{i}$, we first exploit voxel query to group a set of neighboring voxels $\Gamma\_{i}=\left$\mathbf{v}\_{i}^{1}, \mathbf{v}\_{i}^{2}, \cdots, \mathbf{v}\_{i}^{K}\right$ .$ Then, we aggregate the neighboring voxel features with a [PointNet](https://paperswithcode.com/method/pointnet) module $\mathrm{a}$ as:

$$
\mathbf{\eta}\_{i}=\max _{k=1,2, \cdots, K}\left$\Psi\left(\left[\mathbf{v}\_{i}^{k}-\mathbf{g}\_{i} ; \mathbf{\phi}\_{i}^{k}\right]\right)\right$
$$

where $\mathbf{v}\_{i}-\mathbf{g}\_{i}$ represents the relative coordinates, $\mathbf{\phi}\_{i}^{k}$ is the voxel feature of $\mathbf{v}\_{i}^{k}$, and $\Psi(\cdot)$ indicates an MLP. The [max pooling](https://paperswithcode.com/method/max-pooling) operation $\max (\cdot)$ is performed along the channels to obtain the aggregated feature vector $\eta_{i} .$ Particularly, Voxel RoI pooling is exploited to extract voxel features from the 3D feature volumes out of the last two stages in the $3 \mathrm{D}$ backbone network. And for each stage, two Manhattan distance thresholds are set to group voxels with multiple scales. Then, we concatenate the aggregated features pooled from different stages and scales to obtain the RoI features.

Voxel RoI Pooling

Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection

Please enter a description about the method here

ooJpiued

Dynamic Network Model from Partial Observations

Smooth Step

The Tree Ensemble Layer: Differentiability meets Conditional Computation

**Hit-Detector** is a neural architectures search algorithm that simultaneously searches all components of an object detector in an end-to-end manner. It is a hierarchical approach to mine the proper subsearch space from the large volume of operation candidates. It consists of two main procedures. First, given a large search space containing all the operation candidates, we screen out the customized sub search space suitable for each part of detector with the help of group sparsity regularization. Secondly, we search the architectures for each part within the corresponding sub search space by adopting the differentiable manner.

Hit-Detector

Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection

PGC-DGCNN provides a new definition of graph convolutional filter. It generalizes the most commonly adopted filter, adding an hyper-parameter controlling the distance of the considered neighborhood. The model extends graph convolutions, following an intuition derived from the well-known convolutional filters over multi-dimensional tensors. The methods involves a simple, efficient and effective way to introduce a hyper-parameter on graph convolutions that influences the filter size, i.e. its receptive field over the considered graph.

Description and image from: [On Filter Size in Graph Convolutional Networks](https://arxiv.org/pdf/1811.10435.pdf)

PGC-DGCNN

On Filter Size in Graph Convolutional Networks

**RealFormer** is a type of [Transformer](https://paperswithcode.com/methods/category/transformers) based on the idea of [residual](https://paperswithcode.com/method/residual-connection) attention. It adds skip edges to the backbone [Transformer](https://paperswithcode.com/method/transformer) to create multiple direct paths, one for each type of attention module. It adds no parameters or hyper-parameters. Specifically, RealFormer uses a Post-[LN](https://paperswithcode.com/method/layer-normalization) style Transformer as backbone and adds skip edges to connect [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention) modules in adjacent layers.

AI Glossary