What is: AdaMax?

AdaMax is a generalisation of Adam from the $l\_{2}$ norm to the $l\_{\infty}$ norm. Define:

$u\_{t} = \beta^{\infty}\_{2}v\_{t-1} + \left(1-\beta^{\infty}\_{2}\right)|g\_{t}|^{\infty}$

$= \max\left(\beta\_{2}\cdot{v}\_{t-1}, |g\_{t}|\right)$

We can plug into the Adam update equation by replacing $\sqrt{\hat{v}_{t} + \epsilon}$ with $u\_{t}$ to obtain the AdaMax update rule:

$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{u\_{t}}\hat{m}\_{t}$

Common default values are $\eta = 0.002$ and $\beta\_{1}=0.9$ and $\beta\_{2}=0.999$.