What is: AdaSqrt?

AdaSqrt is a stochastic optimization technique that is motivated by the observation that methods like Adagrad and Adam can be viewed as relaxations of Natural Gradient Descent.

The updates are performed as follows:

$t \leftarrow t + 1$

$\alpha\_{t} \leftarrow \sqrt{t}$

$g\_{t} \leftarrow \nabla\_{\theta}f\left(\theta\_{t-1}\right)$

$S\_{t} \leftarrow S\_{t-1} + g\_{t}^{2}$

$\theta\_{t+1} \leftarrow \theta\_{t} + \eta\frac{\alpha\_{t}g\_{t}}{S\_{t} + \epsilon}$