Viet-Anh on Software Logo

What is: Residual Normal Distribution?

SourceNVAE: A Deep Hierarchical Variational Autoencoder
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Residual Normal Distributions are used to help the optimization of VAEs, preventing optimization from entering an unstable region. This can happen due to sharp gradients caused in situations where the encoder and decoder produce distributions far away from each other. The residual distribution parameterizes q(zx)q\left(\mathbf{z}|\mathbf{x}\right) relative to p(z)p\left(\mathbf{z}\right). Let p(zi_lz_<l):=N(μ_i(z_<l),σ_i(z_<l))p\left(z^{i}\_{l}|\mathbf{z}\_{<l}\right) := N \left(\mu\_{i}\left(\mathbf{z}\_{<l}\right), \sigma\_{i}\left(\mathbf{z}\_{<l}\right)\right) be a Normal distribution for the iith variable in z_l\mathbf{z}\_{l} in prior. Define q(zi_lz_<l,x):=N(μ_i(z_<l)+Δμ_i(z_<l,x),σ_i(z_<l)Δσ_i(z_<l,x))q\left(z^{i}\_{l}|\mathbf{z}\_{<l}, x\right) := N\left(\mu\_{i}\left(\mathbf{z}\_{<l}\right) + \Delta\mu\_{i}\left(\mathbf{z}\_{<l}, x\right), \sigma\_{i}\left(\mathbf{z}\_{<l}\right) \cdot \Delta\sigma\_{i}\left(\mathbf{z}\_{<l}, x\right) \right), where Δμ_i(z_<l,x)\Delta\mu\_{i}\left(\mathbf{z}\_{<l}, \mathbf{x}\right) and Δσ_i(z_<l,x)\Delta\sigma\_{i}\left(\mathbf{z}\_{<l}, \mathbf{x}\right) are the relative location and scale of the approximate posterior with respect to the prior. With this parameterization, when the prior moves, the approximate posterior moves accordingly, if not changed.