**Movement Pruning** is a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. Magnitude pruning can be seen as utilizing zeroth-order information (absolute value) of the running model. In contrast, movement pruning methods are where importance is derived from first-order information. Intuitively, instead of selecting weights that are far from zero, we retain connections that are moving away from zero during the training process.

**Contextual Residual Aggregation**, or **CRA**, is a module for image inpainting. It can produce high-frequency residuals for missing contents by weighted aggregating residuals from contextual patches, thus only requiring a low-resolution prediction from the network. Specifically, it involves a neural network to predict a low-resolution inpainted result and up-sample it to yield a large blurry image. Then we produce the high-frequency residuals for in-hole patches by aggregating weighted high-frequency residuals from contextual patches. Finally, we add the aggregated residuals to the large blurry image to obtain a sharp result.

Contextual Residual Aggregation

Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting

Movement Pruning

Movement Pruning: Adaptive Sparsity by Fine-Tuning

The method exploits the finding that high correlation of segmentation performance among each U-Net's decoder layer -- with discriminative layer attached -- tends to have higher segmentation performance in the final segmentation map. By introducing an "Inter-layer Divergence Loss", based on Kulback-Liebler Divergence, to promotes the consistency between each discriminative output from decoder layers by minimizing the divergence.

If we assume that each decoder layer is equivalent to PDE function parameterized by weight parameter $\theta$:

$Decoder_i(x;\theta_i) \equiv PDE(x;\theta_i)$

Then our objective is trying to make each discriminative output similar to each other:

$PDE(x; \theta_d) \sim PDE(x; \theta_i);\text{ } 0 \leq i < d$

Hence the objective is to $\text{minimize} \sum_{i=0}^{d} D_{KL}(\hat{y} || Decoder_i)$.

Source	Movement Pruning: Adaptive Sparsity by Fine-Tuning
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com