**Slow Momentum** (SlowMo) is a distributed optimization method where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm.  Periodically, after taking some number $\tau$ of base algorithm steps, workers average their parameters using ALLREDUCE and perform a momentum update.

**TopK Copy** is a cross-attention guided copy mechanism for entity extraction where only the Top-$k$ important attention heads are used for computing copy distributions. The motivation is that that attention heads may not equally important, and that some heads can be pruned out with a marginal decrease in overall performance. Attention probabilities produced by insignificant attention heads may be noisy. Thus, computing copy distributions without these heads could improve the model’s ability to infer the importance of each token in the input document.

TopK Copy

Document-level Entity-based Extraction as Template Generation

SlowMo

SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

Characterizable Invertible $3\times3$  Convolution

Source	SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com