**Nesterov Accelerated Gradient** is a momentum-based [SGD](https://paperswithcode.com/method/sgd) optimizer that "looks ahead" to where the parameters will be to calculate the gradient **ex post** rather than **ex ante**:

$$ v\_{t} = \gamma{v}\_{t-1} + \eta\nabla\_{\theta}J\left(\theta-\gamma{v\_{t-1}}\right) $$
$$\theta\_{t} = \theta\_{t-1} + v\_{t}$$

Like SGD with momentum $\gamma$ is usually set to $0.9$.

The intuition is that the [standard momentum](https://paperswithcode.com/method/sgd-with-momentum) method first computes the gradient at the current location and then takes a big jump in the direction of the updated accumulated gradient. In contrast Nesterov momentum first makes a big jump in the direction of the previous accumulated gradient and then measures the gradient where it ends up and makes a correction. The idea being that it is better to correct a mistake after you have made it. 

Image Source: [Geoff Hinton lecture notes](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)

**Crossbow** is a single-server multi-GPU system for training deep learning models that enables users to freely choose their preferred batch size—however small—while scaling to multiple GPUs. Crossbow uses many parallel model replicas and avoids reduced statistical efficiency through a new synchronous training method. [SMA](https://paperswithcode.com/method/slime-mould-algorithm-sma), a synchronous variant of model averaging, is used in which replicas independently explore the solution space with gradient descent, but adjust their search synchronously based on the trajectory of a globally-consistent average model.

Crossbow

CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers

Nesterov Accelerated Gradient

A **Dutch Eligibility Trace** is a type of [eligibility trace](https://paperswithcode.com/method/eligibility-trace) where the trace increments grow less quickly than the accumulative eligibility trace (helping avoid large variance updates). For the memory vector $\textbf{e}\_{t} \in \mathbb{R}^{b} \geq \textbf{0}$:

$$\mathbf{e\_{0}} = \textbf{0}$$

$$\textbf{e}\_{t} = \gamma\lambda\textbf{e}\_{t-1} + \left(1-\alpha\gamma\lambda\textbf{e}\_{t-1}^{T}\phi\_{t}\right)\phi\_{t}$$

Year	1983
Data Source	CC BY-SA - https://paperswithcode.com