The method exploits the finding that high correlation of segmentation performance among each U-Net's decoder layer -- with discriminative layer attached -- tends to have higher segmentation performance in the final segmentation map. By introducing an "Inter-layer Divergence Loss", based on Kulback-Liebler Divergence, to promotes the consistency between each discriminative output from decoder layers by minimizing the divergence.

If we assume that each decoder layer is equivalent to PDE function parameterized by weight parameter $\theta$:

$Decoder_i(x;\theta_i) \equiv PDE(x;\theta_i)$

Then our objective is trying to make each discriminative output similar to each other:

$PDE(x; \theta_d) \sim PDE(x; \theta_i);\text{ } 0 \leq i < d$

Hence the objective is to $\text{minimize} \sum_{i=0}^{d} D_{KL}(\hat{y} || Decoder_i)$.

**Movement Pruning** is a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. Magnitude pruning can be seen as utilizing zeroth-order information (absolute value) of the running model. In contrast, movement pruning methods are where importance is derived from first-order information. Intuitively, instead of selecting weights that are far from zero, we retain connections that are moving away from zero during the training process.

Movement Pruning

Movement Pruning: Adaptive Sparsity by Fine-Tuning

SSDS

OCTAve: 2D en face Optical Coherence Tomography Angiography Vessel Segmentation in Weakly-Supervised Learning with Locality Augmentation

**Feature Pyramid Grids**, or **FPG**, is a deep multi-pathway feature pyramid, that represents the feature scale-space as a regular grid of parallel bottom-up pathways which are fused by multi-directional lateral connections. It connects the backbone features, $C$, of a ConvNet with a regular structure of $p$ parallel top-down pyramid pathways which are fused by multi-directional lateral connections, AcrossSame, AcrossUp, AcrossDown, and AcrossSkip. AcrossSkip are direct connections while all other types use [convolutional](https://paperswithcode.com/method/convolution) and [ReLU](https://paperswithcode.com/method/relu) layers.

On a high-level, FPG is a deep generalization of [FPN](https://paperswithcode.com/method/fpn) from one to $p$ pathways under a dense lateral connectivity structure.

Source	OCTAve: 2D en face Optical Coherence Tomography Angiography Vessel Segmentation in Weakly-Supervised Learning with Locality Augmentation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com