Viet-Anh on Software Logo

What is: SlowMo?

SourceSlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Slow Momentum (SlowMo) is a distributed optimization method where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm. Periodically, after taking some number τ\tau of base algorithm steps, workers average their parameters using ALLREDUCE and perform a momentum update.