Please enter a description about the method here

A **Scale Aggregation Block** concatenates feature maps at a wide range of scales. Feature maps for each scale are generated by a stack of downsampling, [convolution](https://paperswithcode.com/method/convolution) and upsampling operations. The proposed scale aggregation block is a standard computational module which readily replaces any given transformation $\mathbf{Y}=\mathbf{T}(\mathbf{X})$, where $\mathbf{X}\in \mathbb{R}^{H\times W\times C}$, $\mathbf{Y}\in \mathbb{R}^{H\times W\times C_o}$ with $C$ and $C_o$ being the input and output channel number respectively. $\mathbf{T}$ is any operator such as a convolution layer or a series of convolution layers. Assume we have $L$ scales. Each scale $l$ is generated by sequentially conducting a downsampling $\mathbf{D}_l$, a transformation $\mathbf{T}_l$ and an unsampling operator $\mathbf{U}_l$:

$$
\mathbf{X}^{'}_l=\mathbf{D}_l(\mathbf{X}),
\label{eq:eq_d}
$$

$$
\mathbf{Y}^{'}_l=\mathbf{T}_l(\mathbf{X}^{'}_l),
\label{eq:eq_tl}
$$

$$
\mathbf{Y}_l=\mathbf{U}_l(\mathbf{Y}^{'}_l),
\label{eq:eq_u}
$$

where $\mathbf{X}^{'}_l\in \mathbb{R}^{H_l\times W_l\times C}$,
$\mathbf{Y}^{'}_l\in \mathbb{R}^{H_l\times W_l\times C_l}$, and
$\mathbf{Y}_l\in \mathbb{R}^{H\times W\times C_l}$.
Notably, $\mathbf{T}_l$ has the similar structure as $\mathbf{T}$.
We can concatenate all $L$ scales together, getting

$$
\mathbf{Y}^{'}=\Vert^L_1\mathbf{U}_l(\mathbf{T}_l(\mathbf{D}_l(\mathbf{X}))),
\label{eq:eq_all}
$$

where $\Vert$ indicates concatenating feature maps along the channel dimension, and $\mathbf{Y}^{'} \in \mathbb{R}^{H\times W\times \sum^L_1 C_l}$ is the final output feature maps of the scale aggregation block.

In the reference implementation, the downsampling $\mathbf{D}_l$ with factor $s$ is implemented by a max pool layer with $s\times s$ kernel size and  $s$ stride. The upsampling $\mathbf{U}_l$ is implemented by resizing with the nearest neighbor  interpolation.

Scale Aggregation Block

Data-Driven Neuron Allocation for Scale Aggregation Networks

Targeted Dropout

Learning Sparse Networks Using Targeted Dropout

**Aggregated Momentum (AggMo)** is a variant of the [classical momentum](https://paperswithcode.com/method/sgd-with-momentum) stochastic optimizer which maintains several velocity vectors with different $\beta$ parameters. AggMo averages the velocity vectors when updating the parameters. It resolves the problem of choosing a momentum parameter by taking a linear combination of multiple momentum buffers. Each of $K$ momentum buffers have a different discount factor $\beta \in \mathbb{R}^{K}$, and these are averaged for the update. The update rule is:

$$ \textbf{v}\_{t}^{\left(i\right)} = \beta^{(i)}\textbf{v}\_{t-1}^{\left(i\right)} - \nabla\_{\theta}f\left(\mathbf{\theta}\_{t-1}\right) $$

$$ \mathbf{\theta\_{t}} = \mathbf{\theta\_{t-1}} + \frac{\gamma\_{t}}{K}\sum^{K}\_{i=1}\textbf{v}\_{t}^{\left(i\right)} $$

where $v^{\left(i\right)}_{0}$ for each $i$. The vector $\mathcal{\beta} = \left[\beta^{(1)}, \ldots, \beta^{(K)}\right]$ is the dampening factor.

Source	Learning Sparse Networks Using Targeted Dropout
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com