**Temporal Activation Regularization (TAR)** is a type of slowness regularization for [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks) that penalizes differences between states that have been explored in the past. Formally we minimize:

$$\beta{L\_{2}}\left(h\_{t} - h\_{t+1}\right)$$

where $L\_{2}$ is the $L\_{2}$ norm, $h_{t}$ is the output of the RNN at timestep $t$, and $\beta$ is a scaling coefficient.

The **Shuffle Transformer Block** consists of the Shuffle Multi-Head Self-Attention module (ShuffleMHSA), the Neighbor-Window Connection module (NWC), and the MLP module. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, a strategy which alternates between WMSA and Shuffle-WMSA in consecutive Shuffle Transformer blocks is proposed. The first window-based transformer block uses regular window partition strategy and the second window-based transformer block uses window-based selfattention with spatial shuffle. Besides, the Neighbor-Window Connection moduel (NWC) is added into each block for enhancing connections among neighborhood windows. Thus the proposed shuffle transformer block could build rich cross-window connections and augments representation. Finally, the consecutive Shuffle Transformer blocks are computed as:

$$ x^{l}=\mathbf{W M S A}\left(\mathbf{B N}\left(z^{l-1}\right)\right)+z^{l-1} $$

$$ y^{l}=\mathbf{N W C}\left(x^{l}\right)+x^{l} $$

$$ z^{l}=\mathbf{M L P}\left(\mathbf{B N}\left(y^{l}\right)\right)+y^{l} $$

$$ x^{l+1}=\mathbf{S h u f f l e - W M S A}\left(\mathbf{B N}\left(z^{l}\right)\right)+z^{l} $$

$$ y^{l+1}=\mathbf{N W C}\left(x^{l+1}\right)+x^{l+1} $$

$$ z^{l+1}=\mathbf{M L P}\left(\mathbf{B N}\left(y^{l+1}\right)\right)+y^{l+1} $$

where $x^l$, $y^l$ and $z^l$ denote the output features of the (Shuffle-)WMSA module, the Neighbor-Window Connection module and the MLP module for block $l$, respectively; WMSA and Shuffle-WMSA denote
window-based multi-head self-attention without/with spatial shuffle, respectively.

Shuffle-T

Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer

Temporal Activation Regularization

Revisiting Activation Regularization for Language RNNs

Non-maximum suppression is an integral part of the object detection pipeline. First, it sorts all detection boxes on the basis of their scores. The detection box $M$ with the maximum score is selected and all other detection boxes with a significant overlap (using a pre-defined threshold)
with $M$ are suppressed. This process is recursively applied on the remaining boxes. As per the design of the algorithm, if an object lies within the predefined overlap threshold, it leads to a miss. 

**Soft-NMS** solves this problem by decaying the detection scores of all other objects as a continuous function of their overlap with M. Hence, no object is eliminated in this process.

Source	Revisiting Activation Regularization for Language RNNs
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com