A **Non-Local Block** is an image block module used in neural networks that wraps a [non-local operation](https://paperswithcode.com/method/non-local-operation). We can define a non-local block as:

$$ \mathbb{z}\_{i} = W\_{z}\mathbb{y\_{i}} + \mathbb{x}\_{i} $$

where $y\_{i}$ is the output from the non-local operation and $+ \mathbb{x}\_{i}$ is a [residual connection](https://paperswithcode.com/method/residual-connection).

**Randomized Leaky Rectified Linear Units**, or **RReLU**, are an activation function that randomly samples the negative slope for activation values. It was first proposed and used in the Kaggle NDSB Competition. During training, $a\_{ji}$ is a random number sampled from a uniform distribution $U\left(l, u\right)$. Formally:

$$ y\_{ji} = x\_{ji} \text{   if } x\_{ji} \geq{0} $$
$$ y\_{ji} = a\_{ji}x\_{ji} \text{   if } x\_{ji} < 0 $$

where

$$\alpha\_{ji} \sim U\left(l, u\right), l < u \text{ and } l, u \in \left[0,1\right)$$

In the test phase, we take average of all the $a\_{ji}$ in training similar to [dropout](https://paperswithcode.com/method/dropout), and thus set $a\_{ji}$ to $\frac{l+u}{2}$ to get a deterministic result. As suggested by the NDSB competition winner, $a\_{ji}$ is sampled from $U\left(3, 8\right)$. 

At test time, we use:

$$ y\_{ji} = \frac{x\_{ji}}{\frac{l+u}{2}} $$

RReLU

Empirical Evaluation of Rectified Activations in Convolutional Network

Non-Local Block

Non-local Neural Networks

**CenterMask** is an anchor-free instance segmentation method that adds a novel [spatial attention-guided mask](https://paperswithcode.com/method/spatial-attention-guided-mask) (SAG-Mask) branch to anchor-free one stage object detector ([FCOS](https://paperswithcode.com/method/fcos)) in the same vein with [Mask R-CNN](https://paperswithcode.com/method/mask-r-cnn). Plugged into the FCOS object detector, the SAG-Mask branch predicts a segmentation mask on each detected box with the spatial attention map that helps to focus on informative pixels and suppress noise.

Source	Non-local Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com