Viet-Anh on Software Logo

What is: LARS?

SourceLarge Batch Training of Convolutional Networks
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Layer-wise Adaptive Rate Scaling, or LARS, is a large batch optimization technique. There are two notable differences between LARS and other adaptive algorithms such as Adam or RMSProp: first, LARS uses a separate learning rate for each layer and not for each weight. And second, the magnitude of the update is controlled with respect to the weight norm for better control of training speed.

m_t=β_1m_t1+(1β_1)(g_t+λx_t)m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)\left(g\_{t} + \lambda{x\_{t}}\right) x_t+1(i)=x_t(i)η_tϕ(x_t(i))m_t(i)m_t(i)x\_{t+1}^{\left(i\right)} = x\_{t}^{\left(i\right)} - \eta\_{t}\frac{\phi\left(|| x\_{t}^{\left(i\right)} ||\right)}{|| m\_{t}^{\left(i\right)} || }m\_{t}^{\left(i\right)}