Viet-Anh on Software Logo

What is: Polyak Averaging?

Year1991
Data SourceCC BY-SA - https://paperswithcode.com

Polyak Averaging is an optimization technique that sets final parameters to an average of (recent) parameters visited in the optimization trajectory. Specifically if in tt iterations we have parameters θ_1,θ_2,,θ_t\theta\_{1}, \theta\_{2}, \dots, \theta\_{t}, then Polyak Averaging suggests setting

θ_t=1t_iθ_i\theta\_t =\frac{1}{t}\sum\_{i}\theta\_{i}

Image Credit: Shubhendu Trivedi & Risi Kondor