Theoretical Foundations of Flow-based Methods for Sampling and Generative Modeling

Abstract

Sampling from an arbitrary probability distribution is a central problem in computational statistics and machine learning. Transportation of measure offers a useful approach to this problem: the idea is to construct a measurable map that pushes forward a relatively simple source distribution to the target probability distribution. One can then simulate from the target distribution by drawing samples from the source distribution and evaluating the transport map. This construction is applicable to both generative modeling and variational inference; when the map is invertible, one can also estimate the density of the target measure by evaluating the density of the pushforward of the source distribution under the inverse transport map. Over the past decade, various parameterizations of such transports have been proposed. Generally speaking, they fall into two categories: the static approach, where the displacement from x to T(x) is represented directly and the dynamic approach that employs evolution of measures by some differential equation over some fictitious time. While many of these models have achieved enormous success in practical applications, their theoretical underpinnings remain largely unexplored. In this thesis, we provide a theoretical foundation for flow-based methods for sampling and generative modeling, and a unified view of both continuous and discrete-time approaches. In the first part of the thesis, we address the approximation theory of flow based methods. In particular, we show how the regularity of the underlying ODE velocity field relates to the regularity of densities and prove related neural network approximation bounds. In addition, we show how introduction of a time-reparameterized schedule can dramatically improve the regularity of the velocity, helping resolve potential singularities. In the second part of the thesis, we focus on the interplay between flow-based models and nonparametric statistics. In particular, we consider pullback density estimators under these flow based models from likelihood-based objectives. The estimators we consider arise from both discrete and continuous-time parameterizations of the transport, and the underlying function classes we consider include Hölder balls and neural networks. In all these cases, we show they achieve near minimax optimal rate for learning s-smooth densities.