Viet-Anh on Software Logo

What is: Normalizing Flows?

SourceVariational Inference with Normalizing Flows
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Normalizing Flows are a method for constructing complex distributions by transforming a probability density through a series of invertible mappings. By repeatedly applying the rule for change of variables, the initial density ‘flows’ through the sequence of invertible mappings. At the end of this sequence we obtain a valid probability distribution and hence this type of flow is referred to as a normalizing flow.

In the case of finite flows, the basic rule for the transformation of densities considers an invertible, smooth mapping f:RdRdf : \mathbb{R}^{d} \rightarrow \mathbb{R}^{d} with inverse f1=gf^{-1} = g, i.e. the composition gf(z)=zg \cdot f\left(z\right) = z. If we use this mapping to transform a random variable zz with distribution q(z)q\left(z\right), the resulting random variable z=f(z)z' = f\left(z\right) has a distribution:

q(z)=q(z)detδf1δz=q(z)detδfδz1q\left(\mathbf{z}'\right) = q\left(\mathbf{z}\right)\bigl\vert{\text{det}}\frac{\delta{f}^{-1}}{\delta{\mathbf{z'}}}\bigr\vert = q\left(\mathbf{z}\right)\bigl\vert{\text{det}}\frac{\delta{f}}{\delta{\mathbf{z}}}\bigr\vert ^{-1} where the last equality can be seen by applying the chain rule (inverse function theorem) and is a property of Jacobians of invertible functions. We can construct arbitrarily complex densities by composing several simple maps and successively applying the above equation. The density q_K(z)q\_{K}\left(\mathbf{z}\right) obtained by successively transforming a random variable z_0z\_{0} with distribution q_0q\_{0} through a chain of KK transformations f_kf\_{k} is:

z_K=f_Kf_2f_1(z_0)z\_{K} = f\_{K} \cdot \dots \cdot f\_{2} \cdot f\_{1}\left(z\_{0}\right)

lnq_K(z_K)=lnq_0(z_0)K_k=1lndetδf_kδz_k1\ln{q}\_{K}\left(z\_{K}\right) = \ln{q}\_{0}\left(z\_{0}\right) − \sum^{K}\_{k=1}\ln\vert\det\frac{\delta{f\_{k}}}{\delta{\mathbf{z\_{k-1}}}}\vert The path traversed by the random variables z_k=f_k(z_k1)z\_{k} = f\_{k}\left(z\_{k-1}\right) with initial distribution q_0(z_0)q\_{0}\left(z\_{0}\right) is called the flow and the path formed by the successive distributions q_kq\_{k} is a normalizing flow.