Viet-Anh on Software Logo

What is: Lookahead?

SourceLookahead Optimizer: k steps forward, 1 step back
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Lookahead is a type of stochastic optimizer that iteratively updates two sets of weights: "fast" and "slow". Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of fast weights generated by another optimizer.

Algorithm 1 Lookahead Optimizer

Require Initial parameters ϕ0\phi_0, objective function LL

Require Synchronization period kk, slow weights step size α\alpha, optimizer AA

   for t=1,2,t=1, 2, \dots

     Synchronize parameters θt,0ϕt1\theta_{t,0} \gets \phi_{t-1}

     for i=1,2,,ki=1, 2, \dots, k

       sample minibatch of data dDd \sim \mathcal{D}

       θt,iθt,i1+A(L,θt,i1,d)\theta_{t,i} \gets \theta_{t,i-1} + A(L, \theta_{t,i-1}, d)

     endfor

     Perform outer update ϕtϕt1+α(θt,kϕt1)\phi_t \gets \phi_{t-1} + \alpha (\theta_{t,k} - \phi_{t-1})

   endfor

   return parameters ϕ\phi