**Activation Regularization (AR)**, or $L\_{2}$ activation regularization, is regularization performed on activations as opposed to weights. It is usually used in conjunction with [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks). It is defined as:

$$\alpha{L}\_{2}\left(m\circ{h\_{t}}\right) $$

where $m$ is a [dropout](https://paperswithcode.com/method/dropout) mask used by later parts of the model, $L\_{2}$ is the $L\_{2}$ norm, and $h_{t}$ is the output of an RNN at timestep $t$, and $\alpha$ is a scaling coefficient. 

When applied to the output of a dense layer, AR penalizes activations that are substantially away from 0, encouraging activations to remain small.

AugMix mixes augmented images through linear interpolations. Consequently it is like [Mixup](https://paperswithcode.com/method/mixup) but instead mixes augmented versions of the same image.

AugMix

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Activation Regularization

Revisiting Activation Regularization for Language RNNs

**SepFormer** is [Transformer](https://paperswithcode.com/methods/category/transformers)-based neural network for speech separation. The SepFormer learns short and long-term dependencies with a multi-scale approach that employs transformers. It is mainly composed of multi-head attention and feed-forward layers. A dual-path framework (introduced by DPRNN) is adopted and [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks) are replaced with a multiscale pipeline composed of transformers that learn both short and long-term dependencies. The dual-path framework enables the mitigation of the quadratic complexity of transformers, as transformers in the dual-path framework process smaller chunks.

The model is based on the learned-domain masking approach and employs an encoder, a decoder, and a masking network, as shown in the figure. The encoder is fully convolutional, while the decoder employs two Transformers embedded inside the dual-path processing block. The decoder finally reconstructs the separated signals in the time domain by using the masks predicted by the masking network.

Source	Revisiting Activation Regularization for Language RNNs
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com