Viet-Anh on Software Logo

What is: AggMo?

SourceAggregated Momentum: Stability Through Passive Damping
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Aggregated Momentum (AggMo) is a variant of the classical momentum stochastic optimizer which maintains several velocity vectors with different β\beta parameters. AggMo averages the velocity vectors when updating the parameters. It resolves the problem of choosing a momentum parameter by taking a linear combination of multiple momentum buffers. Each of KK momentum buffers have a different discount factor βRK\beta \in \mathbb{R}^{K}, and these are averaged for the update. The update rule is:

v_t(i)=β(i)v_t1(i)_θf(θ_t1)\textbf{v}\_{t}^{\left(i\right)} = \beta^{(i)}\textbf{v}\_{t-1}^{\left(i\right)} - \nabla\_{\theta}f\left(\mathbf{\theta}\_{t-1}\right)

θ_t=θ_t1+γ_tKK_i=1v_t(i)\mathbf{\theta\_{t}} = \mathbf{\theta\_{t-1}} + \frac{\gamma\_{t}}{K}\sum^{K}\_{i=1}\textbf{v}\_{t}^{\left(i\right)}

where v0(i)v^{\left(i\right)}_{0} for each ii. The vector β=[β(1),,β(K)]\mathcal{\beta} = \left[\beta^{(1)}, \ldots, \beta^{(K)}\right] is the dampening factor.