Viet-Anh on Software Logo

What is: Local Response Normalization?

SourceImageNet Classification with Deep Convolutional Neural Networks
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Local Response Normalization is a normalization layer that implements the idea of lateral inhibition. Lateral inhibition is a concept in neurobiology that refers to the phenomenon of an excited neuron inhibiting its neighbours: this leads to a peak in the form of a local maximum, creating contrast in that area and increasing sensory perception. In practice, we can either normalize within the same channel or normalize across channels when we apply LRN to convolutional neural networks.

bc=ac(k+αnc=max(0,cn/2)min(N1,c+n/2)ac2)βb_{c} = a_{c}\left(k + \frac{\alpha}{n}\sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta}

Where the size is the number of neighbouring channels used for normalization, α\alpha is multiplicative factor, β\beta an exponent and kk an additive factor