Viet-Anh on Software Logo

What is: Spectral-Normalized Identity Priors?

SourcePruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Spectral-Normalized Identity Priors, or SNIP, is a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. It is applicable to any structured module, including a single attention head, an entire attention block, or a feed-forward subnetwork. The method identifies and discards unimportant non-linear mappings in the residual connections by applying a thresholding operator on the function norm. Furthermore, spectral normalization to stabilize the distribution of the post-activation values of the Transformer layers, further improving the pruning effectiveness of the proposed methodology.