**Lookahead** is a type of stochastic optimizer that iteratively updates two sets of weights: "fast" and "slow". Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of *fast weights* generated by another optimizer.



**Algorithm 1** Lookahead Optimizer

**Require** Initial parameters $\phi_0$, objective function $L$ 

**Require** Synchronization period $k$, slow weights step size $\alpha$, optimizer $A$

&nbsp;&nbsp;  **for** $t=1, 2, \dots$

&nbsp;&nbsp;&nbsp;&nbsp; Synchronize parameters $\theta_{t,0} \gets \phi_{t-1}$

&nbsp;&nbsp;&nbsp;&nbsp; **for** $i=1, 2, \dots, k$

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sample minibatch of data $d \sim \mathcal{D}$

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\theta_{t,i} \gets \theta_{t,i-1} + A(L, \theta_{t,i-1}, d)$

&nbsp;&nbsp;&nbsp;&nbsp; **endfor**

&nbsp;&nbsp;&nbsp;&nbsp; Perform outer update $\phi_t \gets \phi_{t-1} + \alpha (\theta_{t,k} - \phi_{t-1})$

&nbsp;&nbsp; **endfor**

&nbsp;&nbsp; **return** parameters $\phi$

A **Dense Block** is a module used in convolutional neural networks that connects *all layers* (with matching feature-map sizes) directly with each other. It was originally proposed as part of the [DenseNet](https://paperswithcode.com/method/densenet) architecture. To preserve the feed-forward nature, each layer obtains additional inputs from all preceding layers and passes on its own feature-maps to all subsequent layers. In contrast to [ResNets](https://paperswithcode.com/method/resnet), we never combine features through summation before they are passed into a layer; instead, we combine features by concatenating them. Hence, the $\ell^{th}$ layer has $\ell$ inputs, consisting of the feature-maps of all preceding convolutional blocks. Its own feature-maps are passed on to all $L-\ell$ subsequent layers. This introduces $\frac{L(L+1)}{2}$  connections in an $L$-layer network, instead of just $L$, as in traditional architectures: "dense connectivity".

Dense Block

Densely Connected Convolutional Networks

Lookahead

Lookahead Optimizer: k steps forward, 1 step back

**Grid R-CNN** is an object detection framework, where the traditional regression
formulation is replaced by a grid point guided localization mechanism.

Grid R-CNN divides the object bounding box region into grids and employs a fully convolutional network ([FCN](https://paperswithcode.com/method/fcn)) to predict the locations of grid points. Owing to the position sensitive property of fully convolutional architecture, Grid R-CNN maintains the explicit spatial information and grid points locations can be obtained in pixel level. When a certain number of grid points at specified location are known, the corresponding bounding box is definitely determined. Guided by the grid points, Grid R-CNN can determine more accurate object bounding box than regression method which lacks the guidance of explicit spatial information.

Source	Lookahead Optimizer: k steps forward, 1 step back
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com