Viet-Anh on Software Logo

What is: BatchChannel Normalization?

SourceRethinking Normalization and Elimination Singularity in Neural Networks
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Batch-Channel Normalization, or BCN, uses batch knowledge to prevent channel-normalized models from getting too close to "elimination singularities". Elimination singularities correspond to the points on the training trajectory where neurons become consistently deactivated. They cause degenerate manifolds in the loss landscape which will slow down training and harm model performances.