**Crossbow** is a single-server multi-GPU system for training deep learning models that enables users to freely choose their preferred batch size—however small—while scaling to multiple GPUs. Crossbow uses many parallel model replicas and avoids reduced statistical efficiency through a new synchronous training method. [SMA](https://paperswithcode.com/method/slime-mould-algorithm-sma), a synchronous variant of model averaging, is used in which replicas independently explore the solution space with gradient descent, but adjust their search synchronously based on the trajectory of a globally-consistent average model.

**LayerDrop** is a form of structured [dropout](https://paperswithcode.com/method/dropout) for [Transformer](https://paperswithcode.com/method/transformer) models which has a regularization effect during training and allows for efficient pruning at inference time. It randomly drops layers from the Transformer according to an "every other" strategy where pruning with a rate $p$ means dropping the layers at depth $d$ such that $d = 0\left\(\text{mod}\left(\text{floor}\left(\frac{1}{p}\right)\right)\right)$.

LayerDrop

Reducing Transformer Depth on Demand with Structured Dropout

Crossbow

CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers

**Nesterov Accelerated Gradient** is a momentum-based [SGD](https://paperswithcode.com/method/sgd) optimizer that "looks ahead" to where the parameters will be to calculate the gradient **ex post** rather than **ex ante**:

$$ v\_{t} = \gamma{v}\_{t-1} + \eta\nabla\_{\theta}J\left(\theta-\gamma{v\_{t-1}}\right) $$
$$\theta\_{t} = \theta\_{t-1} + v\_{t}$$

Like SGD with momentum $\gamma$ is usually set to $0.9$.

The intuition is that the [standard momentum](https://paperswithcode.com/method/sgd-with-momentum) method first computes the gradient at the current location and then takes a big jump in the direction of the updated accumulated gradient. In contrast Nesterov momentum first makes a big jump in the direction of the previous accumulated gradient and then measures the gradient where it ends up and makes a correction. The idea being that it is better to correct a mistake after you have made it. 

Image Source: [Geoff Hinton lecture notes](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)

Source	CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com