Jump to content

Flow-based generative model

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by Citation bot (talk | contribs) at 02:14, 25 July 2022 (Alter: url, template type. URLs might have been anonymized. Add: eprint, class, s2cid, arxiv, pmid, authors 1-1. Removed parameters. Some additions/deletions were parameter name changes. | Use this bot. Report bugs. | Suggested by Headbomb | #UCB_toolbar). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

A flow-based generative model is a generative model used in machine learning that explicitly models a probability distribution by leveraging normalizing flow,[1] which is a statistical method using the change-of-variable law of probabilities to transform a simple distribution into a complex one.

The direct modeling of likelihood provides many advantages. For example, the negative log-likelihood can be directly computed and minimized as the loss function. Additionally, novel samples can be generated by sampling from the initial distribution, and applying the flow transformation.

In contrast, many alternative generative modeling methods such as variational autoencoder (VAE) and generative adversarial network do not explicitly represent the likelihood function.

Method

Scheme for normalizing flows

Let be a (possibly multivariate) random variable with distribution .

For , let be a sequence of random variables transformed from . The functions should be invertible, i.e. the inverse function exists. The final output models the target distribution.

The log likelihood of is (see derivation):

To efficiently compute the log likelihood, the functions should be 1. easy to invert, and 2. easy to compute the determinant of its Jacobian. In practice, the functions are modeled using deep neural networks, and are trained to minimize the negative log-likelihood of data samples from the target distribution. These architectures are usually designed such that only the forward pass of the neural network is required in both the inverse and the Jacobian determinant calculations. Examples of such architectures include NICE,[2] RealNVP,[3] and Glow.[4]

Derivation of log likelihood

Consider and . Note that .

By the change of variable formula, the distribution of is:

Where is the determinant of the Jacobian matrix of .

By the inverse function theorem:

By the identity (where is an invertible matrix), we have:

The log likelihood is thus:

In general, the above applies to any and . Since is equal to subtracted by a non-recursive term, we can infer by induction that:

Training method

Flow-based models are generally trained by maximum likelihood. A pseudocode is as follows:[5]

  • INPUT. dataset , normalizing flow model .
  • SOLVE. by gradient descent
  • RETURN.

Variants

Planar Flow

The earliest example.[1] Fix some activation function , and let with th appropriate dimensions, then

The inverse has no closed-form solution in general.

The Jacobian is .

For it to be invertible everywhere, it must be nonzero everywhere. For example, and satisfies the requirement.

Nonlinear Independent Components Estimation (NICE)

Let be even-dimensional, and split them in the middle.[2] Then the normalizing flow functions are

where is any neural network with weights .

is just , and the Jacobian is just 1, that is, the flow is volume-preserving.

When , this is seen as a curvy shearing along the direction.

Real Non-Volume Preserving (Real NVP)

The Real Non-Volume Preserving model generalizes NICE model by:[3]

Its inverse is , and its Jacobian is . The NICE model is recovered by setting . Since the Real NVP map keeps the first and second halves of the vector separate, it's usually required to add a permutation after every Real NVP layer.

Generative Flow (Glow)

In generative flow model,[4] each layer has 3 parts:

  • channel-wise affine transform
    with Jacobian .
  • invertible 1x1 convolution
    with Jacobian . Here is any invertible matrix.
  • Real NVP, with Jacobian as described in Real NVP.

The idea of using the invertible 1x1 convolution is to permute all layers in general, instead of merely permuting the first and second half, as in Real NVP.

Masked autoregressive flow (MAF)

An autoregressive model of a distribution on is defined as the following stochastic process:[6]

where and are fixed functions that define the autoregressive model.

By the reparametrization trick, the autoregressive model is generalized to a normalizing flow:

The autoregressive model is recovered by setting .

The forward mapping is slow (because it's sequential), but the backward mapping is fast (because it's parallel).

The Jacobian matrix is lower-diagonal, so the Jacobian is .

Reversing the two maps and of MAF results in Inverse Autoregressive Flow (IAF), which has fast forward mapping and slow backward mapping.[7]

Continuous Normalizing Flow (CNF)

Instead of constructing flow by function composition, another approach is to formulate the flow as a continuous-time dynamic.[8] Let be the latent variable with distribution . Map this latent variable to data space with the following flow function:

where is an arbitrary function and can be modeled with e.g. neural networks.

The inverse function is then naturally:[8]

And the log-likelihood of can be found as:[8]

Since the trace depends only on the diagonal of the Jacobian , this allows "free-form" Jacobian.[9] Here, "free-form" means that there is no restriction on the Jacobian's form. It is contrasted with previous discrete models of normalizing flow, where the Jacobian is carefully designed to be only upper- or lower-diagonal, so that the Jacobian can be evaluated efficiently.

The trace can be estimated by "Hutchinson's trick":[10][11]

Given any matrix , and any random with , we have . (Proof: expand the expectation directly.)

Usually, the random vector is sampled from (normal distribution) or (Radamacher distribution).

When is implemented as a neural network, neural ODE methods[12] would be needed. Indeed, CNF was first proposed in the same paper that proposed neural ODE.

There are two main deficiencies of CNF, one is that a continuous flow must be a homeomorphism, thus preserve orientation and ambient isotopy (for example, it's impossible to flip a left-hand to a right-hand by continuous deforming of space, and it's impossible to turn a sphere inside out, or undo a knot), and the other is that the learned flow might be ill-behaved, due to degeneracy (that is, there are an infinite number of possible that all solve the same problem).

By adding extra dimensions, the CNF gains enough freedom to reverse orientation and go beyond ambient isotopy (just like how one can pick up a polygon from a desk and flip it around in 3-space, or unknot a knot in 4-space), yielding the "augmented neural ODE".[13]

Any homeomorphism of can be approximated by a neural ODE operating on , proved by combining Whitney embedding theorem for manifolds and the universal approximation theorem for neural networks.[14]

To regularize the flow , one can impose regularization losses. The paper [10] proposed the following regularization loss based on optimal transport theory:

where are hyperparameters. The first term punishes the model for varying the flow field over time, and the second term punishes it for varying the flow field over space.

Applications

Flow-based generative models have been applied on a variety of modeling tasks, including:

  • Audio generation[15]
  • Image generation[4]
  • Molecular graph generation[16]
  • Point-cloud modeling[17]
  • Video generation[18]

References

  1. ^ a b Danilo Jimenez Rezende; Mohamed, Shakir (2015). "Variational Inference with Normalizing Flows". arXiv:1505.05770 [stat.ML].
  2. ^ a b Dinh, Laurent; Krueger, David; Bengio, Yoshua (2014). "NICE: Non-linear Independent Components Estimation". arXiv:1410.8516 [cs.LG].
  3. ^ a b Dinh, Laurent; Sohl-Dickstein, Jascha; Bengio, Samy (2016). "Density estimation using Real NVP". arXiv:1605.08803 [cs.LG].
  4. ^ a b c Kingma, Diederik P.; Dhariwal, Prafulla (2018). "Glow: Generative Flow with Invertible 1x1 Convolutions". arXiv:1807.03039 [stat.ML].
  5. ^ Kobyzev, Ivan; Prince, Simon J.D.; Brubaker, Marcus A. (November 2021). "Normalizing Flows: An Introduction and Review of Current Methods". IEEE Transactions on Pattern Analysis and Machine Intelligence. 43 (11): 3964–3979. arXiv:1908.09257. doi:10.1109/TPAMI.2020.2992934. ISSN 1939-3539. PMID 32396070. S2CID 208910764.
  6. ^ Papamakarios, George; Pavlakou, Theo; Murray, Iain (2017). "Masked Autoregressive Flow for Density Estimation". Advances in Neural Information Processing Systems. 30. Curran Associates, Inc. arXiv:1705.07057.
  7. ^ Kingma, Durk P; Salimans, Tim; Jozefowicz, Rafal; Chen, Xi; Sutskever, Ilya; Welling, Max (2016). "Improved Variational Inference with Inverse Autoregressive Flow". Advances in Neural Information Processing Systems. 29. Curran Associates, Inc. arXiv:1606.04934.
  8. ^ a b c Grathwohl, Will; Chen, Ricky T. Q.; Bettencourt, Jesse; Sutskever, Ilya; Duvenaud, David (2018). "FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models". arXiv:1810.01367 [cs.LG].
  9. ^ Grathwohl, Will; Chen, Ricky T. Q.; Bettencourt, Jesse; Sutskever, Ilya; Duvenaud, David (2018-10-22). "FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models". arXiv:1810.01367 [cs.LG].
  10. ^ a b Finlay, Chris; Jacobsen, Joern-Henrik; Nurbekyan, Levon; Oberman, Adam (2020-11-21). "How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization". International Conference on Machine Learning. PMLR: 3154–3164. arXiv:2002.02798.
  11. ^ Hutchinson, M.F. (January 1989). "A Stochastic Estimator of the Trace of the Influence Matrix for Laplacian Smoothing Splines". Communications in Statistics - Simulation and Computation. 18 (3): 1059–1076. doi:10.1080/03610918908812806. ISSN 0361-0918.
  12. ^ Chen, Ricky T. Q.; Rubanova, Yulia; Bettencourt, Jesse; Duvenaud, David (2018). "Neural Ordinary Differential Equations". arXiv:1806.07366 [cs.LG].
  13. ^ Dupont, Emilien; Doucet, Arnaud; Teh, Yee Whye (2019). "Augmented Neural ODEs". Advances in Neural Information Processing Systems. 32. Curran Associates, Inc.
  14. ^ Zhang, Han; Gao, Xi; Unterman, Jacob; Arodz, Tom (2019-07-30). "Approximation Capabilities of Neural ODEs and Invertible Residual Networks". arXiv:1907.12998 [cs.LG].
  15. ^ Ping, Wei; Peng, Kainan; Zhao, Kexin; Song, Zhao (2019). "WaveFlow: A Compact Flow-based Model for Raw Audio". arXiv:1912.01219 [cs.SD].
  16. ^ Shi, Chence; Xu, Minkai; Zhu, Zhaocheng; Zhang, Weinan; Zhang, Ming; Tang, Jian (2020). "GraphAF: A Flow-based Autoregressive Model for Molecular Graph Generation". arXiv:2001.09382 [cs.LG].
  17. ^ Yang, Guandao; Huang, Xun; Hao, Zekun; Liu, Ming-Yu; Belongie, Serge; Hariharan, Bharath (2019). "PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows". arXiv:1906.12320 [cs.CV].
  18. ^ Kumar, Manoj; Babaeizadeh, Mohammad; Erhan, Dumitru; Finn, Chelsea; Levine, Sergey; Dinh, Laurent; Kingma, Durk (2019). "VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation". arXiv:1903.01434 [cs.CV].

External links