**YellowFin** is a learning rate and momentum tuner motivated by robustness properties and analysis of quadratic objectives. It stems from a known but obscure fact: the momentum operator's spectral radius is constant in a large subset of the hyperparameter space. For quadratic objectives, the optimizer tunes both the learning rate and the momentum to keep the hyperparameters within a region in which the convergence rate is a constant rate equal to the root momentum. This notion is extended empirically to non-convex objectives. On every iteration, YellowFin optimizes the hyperparameters to minimize a local quadratic optimization.

Weight Reset is an implicit regularization procedure that periodically resets a randomly selected portion of layer weights during the training process, according to predefined probability distributions.

To delineate the Weight Reset procedure, a straightforward formulation is introduced. Assume $\mathcal{B}(p)$ as a multivariate Bernoulli distribution with parameter $p$, and let's propose that $\mathcal{D}$ is an arbitrary distribution used for initializing model weights. At specified intervals (after a certain number of training iterations, except for the last one), a random portion of the weights $W={w^l}$ from selected layers in the neural network undergoes a reset utilizing the following method:
$$
\tilde{w}^l = w^l\cdot (1-m) + \xi\cdot m,
$$
where $\cdot$ operation is an element-wise hadamar type multiplication, $w^l$ are current weights for layer $l$, $\tilde{w}^l$ are reset weights for this layer, $m \sim \mathcal{B}(p^l)$ is a resetting mask, $p^l$ is a resetting rate for a layer $l$, $\xi \sim \mathcal{D}$ are new random weights.

Evidence has indicated that Weight Reset can compete with, and in some instances, surpass traditional regularization techniques.

Given the observable effects of the Weight Reset technique on an increasing number of weights in a model, there's a plausible hypothesis suggesting its potential association with the Double Descent phenomenon.

Weights Reset

The Weights Reset Technique for Deep Neural Networks Implicit Regularization

YellowFin

YellowFin and the Art of Momentum Tuning

**Global and Sliding Window Attention** is an attention pattern for attention-based models. It is motivated by the fact that non-sparse attention in the original [Transformer](https://paperswithcode.com/method/transformer) formulation has a [self-attention component](https://paperswithcode.com/method/scaled) with $O\left(n^{2}\right)$ time and memory complexity where $n$ is the input sequence length and thus, is not efficient to scale to long inputs. 

Since [windowed](https://paperswithcode.com/method/sliding-window-attention) and [dilated](https://paperswithcode.com/method/dilated-sliding-window-attention) attention patterns are not flexible enough to learn task-specific representations, the authors of the [Longformer](https://paperswithcode.com/method/longformer) add “global attention” on few pre-selected input locations. This attention is operation symmetric: that is, a token with a global attention attends to all tokens across the sequence, and all tokens in the sequence attend to it. The Figure to the right shows an example of a sliding window attention with global attention at a few tokens at custom locations. For the example of classification, global attention is used for the [CLS] token, while in the example of Question Answering, global attention is provided on all question tokens.

Source	YellowFin and the Art of Momentum Tuning
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com