Viet-Anh on Software Logo

What is: Weights Reset?

SourceThe Weights Reset Technique for Deep Neural Networks Implicit Regularization
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Weight Reset is an implicit regularization procedure that periodically resets a randomly selected portion of layer weights during the training process, according to predefined probability distributions.

To delineate the Weight Reset procedure, a straightforward formulation is introduced. Assume B(p)\mathcal{B}(p) as a multivariate Bernoulli distribution with parameter pp, and let's propose that D\mathcal{D} is an arbitrary distribution used for initializing model weights. At specified intervals (after a certain number of training iterations, except for the last one), a random portion of the weights W=wlW={w^l} from selected layers in the neural network undergoes a reset utilizing the following method:

w~l=wl(1m)+ξm,\tilde{w}^l = w^l\cdot (1-m) + \xi\cdot m,

where \cdot operation is an element-wise hadamar type multiplication, wlw^l are current weights for layer ll, w~l\tilde{w}^l are reset weights for this layer, mB(pl)m \sim \mathcal{B}(p^l) is a resetting mask, plp^l is a resetting rate for a layer ll, ξD\xi \sim \mathcal{D} are new random weights.

Evidence has indicated that Weight Reset can compete with, and in some instances, surpass traditional regularization techniques.

Given the observable effects of the Weight Reset technique on an increasing number of weights in a model, there's a plausible hypothesis suggesting its potential association with the Double Descent phenomenon.