Viet-Anh on Software Logo

What is: Stochastic Weight Averaging?

SourceAveraging Weights Leads to Wider Optima and Better Generalization
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Stochastic Weight Averaging is an optimization procedure that averages multiple points along the trajectory of SGD, with a cyclical or constant learning rate. On the one hand it averages weights, but it also has the property that, with a cyclical or constant learning rate, SGD proposals are approximately sampling from the loss surface of the network, leading to stochastic weights and helping to discover broader optima.