Viet-Anh on Software Logo

What is: RMSProp?

Year2013
Data SourceCC BY-SA - https://paperswithcode.com

RMSProp is an unpublished adaptive learning rate optimizer proposed by Geoff Hinton. The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:

E[g2]_t=γE[g2]_t1+(1γ)g2_tE\left[g^{2}\right]\_{t} = \gamma E\left[g^{2}\right]\_{t-1} + \left(1 - \gamma\right) g^{2}\_{t}

θ_t+1=θ_tηE[g2]_t+ϵg_t\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]\_{t} + \epsilon}}g\_{t}

Hinton suggests γ=0.9\gamma=0.9, with a good default for η\eta as 0.0010.001.

Image: Alec Radford