ORB-SLAM2 is a complete SLAM system for monocular, stereo and RGB-D cameras, including map reuse, loop closing and relocalization capabilities. The system works in real-time on standard CPUs in a wide variety of environments from small hand-held indoors sequences, to drones flying in industrial environments and cars driving around a city.

Source: [Mur-Artal and Tardos](https://arxiv.org/pdf/1610.06475v2.pdf)

Image source: [Mur-Artal and Tardos](https://arxiv.org/pdf/1610.06475v2.pdf)

**Filter Response Normalization (FRN)** is a type of normalization that combines normalization and an activation function, which can be used as a replacement for other normalizations and activations. It operates on each activation channel of each batch element independently, eliminating the dependency on other batch elements. 

To demonstrate, assume we are dealing with the feed-forward convolutional neural network. We follow the usual convention that the filter responses (activation maps) produced after a [convolution](https://paperswithcode.com/method/convolution) operation are a [4D ](https://paperswithcode.com/method/4d-a)tensor $X$ with shape $[B, W, H, C]$, where $B$ is the mini-batch size, $W, H$ are the spatial extents of the map, and $C$ is the number of filters used in convolution. $C$ is also referred to as output channels. Let $x = X_{b,:,:,c} \in \mathcal{R}^{N}$, where $N = W \times H$, be the vector of filter responses for the $c^{th}$ filter for the $b^{th}$ batch point. 
Let $\nu^2 = \sum\_i x_i^2/N$, be the mean squared norm of $x$. 

Then Filter Response Normalization is defined as the following:

$$
\hat{x} = \frac{x}{\sqrt{\nu^2 + \epsilon}},
$$

where $\epsilon$ is a small positive constant to prevent division by zero.  

A lack of mean centering in FRN can lead to activations having an arbitrary bias away from zero. Such a bias in conjunction with [ReLU](https://paperswithcode.com/method/relu) can have a detrimental effect on learning and lead to poor performance and dead units. To address this the authors augment ReLU with a learned threshold $\tau$ to yield:

$$
z = \max(y, \tau)
$$

Since $\max(y, \tau){=}\max(y-\tau,0){+}\tau{=}\text{ReLU}{(y{-}\tau)}{+}\tau$, the effect of this activation is the same as having a shared bias before and after ReLU.

Filter Response Normalization

Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks

ORB-SLAM2

ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras

**TabTransformer** is a deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. 

As an overview, the architecture comprises a column embedding layer, a stack of $N$ [Transformer](/method/transformer) layers, and a multi-layer perceptron (MLP). The contextual embeddings (outputted by the Transformer layer) are concatenated along with continuous features which is inputted to an MLP. The loss function is then minimized  to learn all the parameters in an end-to-end learning.

Source	ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com