Viet-Anh on Software Logo

What is: Spectral Normalization?

SourceSpectral Normalization for Generative Adversarial Networks
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Spectral Normalization is a normalization technique used for generative adversarial networks, used to stabilize training of the discriminator. Spectral normalization has the convenient property that the Lipschitz constant is the only hyper-parameter to be tuned.

It controls the Lipschitz constant of the discriminator ff by constraining the spectral norm of each layer g:h_inhoutg : \textbf{h}\_{in} \rightarrow \textbf{h}_{out}. The Lipschitz norm g_Lip\Vert{g}\Vert\_{\text{Lip}} is equal to sup_hσ(g(h))\sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right), where σ(a)\sigma\left(a\right) is the spectral norm of the matrix AA (L_2L\_{2} matrix norm of AA):

σ(a)=max_h:h0Ah_2h_2=max_h_21Ah_2\sigma\left(a\right) = \max\_{\textbf{h}:\textbf{h}\neq{0}}\frac{\Vert{A\textbf{h}}\Vert\_{2}}{\Vert\textbf{h}\Vert\_{2}} = \max\_{\Vert\textbf{h}\Vert\_{2}\leq{1}}{\Vert{A\textbf{h}}\Vert\_{2}}

which is equivalent to the largest singular value of AA. Therefore for a linear layer g(h)=Whg\left(\textbf{h}\right) = W\textbf{h} the norm is given by g_Lip=sup_hσ(g(h))=sup_hσ(W)=σ(W)\Vert{g}\Vert\_{\text{Lip}} = \sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right) = \sup\_{\textbf{h}}\sigma\left(W\right) = \sigma\left(W\right) . Spectral normalization normalizes the spectral norm of the weight matrix WW so it satisfies the Lipschitz constraint σ(W)=1\sigma\left(W\right) = 1:

Wˉ_SN(W)=W/σ(W)\bar{W}\_{\text{SN}}\left(W\right) = W / \sigma\left(W\right)