Generative Fractional Diffusion Models (2310.17638v3)

Abstract: We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0,1)$, where $H=0.5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in generative fractional diffusion models (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose augmented score matching to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0.5$ enters the regime of rough paths whereas $H>0.5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models

• The paper introduces generative fractional diffusion models that extend traditional score-based methods by integrating fractional Brownian motion via a weighted sum of OU processes.
• The paper develops a continuous reparameterization trick to bypass Itô calculus limitations, enabling robust modeling of non-Markovian reverse-time dynamics through SDEs.
• The paper validates the approach on toy datasets like Swiss roll and half-moon, demonstrating its capacity to generate samples with varied fidelity based on the Hurst index.

Generative Fractional Diffusion Models

The paper "Generative Fractional Diffusion Models" extends the framework of score-based generative models, traditionally grounded in Brownian motion (BM), to incorporate fractional Brownian motion (FBM). This development is pivotal, as it involves transitioning from stochastic processes with finite quadratic variation to those with infinite quadratic variation. The authors propose generative fractional diffusion models (GFDM) by representing FBM through a stochastic integral over a family of Ornstein-Uhlenbeck (OU) processes. This representation allows GFDM to employ FBM characterized by the Hurst index, $H\in(0,1)$, which modulates the path roughness of the underlying process.

Mathematical Framework

The paper explores a highly mathematical exposition, deriving a continuous reparameterization trick that facilitates the modeling of processes driven by FBM. This trick bypasses the limitations associated with Itô calculus, which is inapplicable to FBM due to its non-Markovian nature and the lack of semimartingale properties. The authors introduce the process $W^{H,m}$, an approximation of FBM as a finite sum of weighted OU processes, ensuring convergence to FBM with arbitrary precision.

The generative model hinges on defining a forward process described by a stochastic differential equation (SDE). This SDE employs $W^{H,m}$ to drive its dynamics, resulting in a Gaussian process where the drift and diffusion terms are parameterized by functions of time. They further extend this to three specific cases: Fractional Variance Exploding (FVE), Fractional Variance Preserving (FVP), and Sub-Fractional Variance Preserving (sub-FVP) processes, each featuring distinct drift and diffusion formulations.

Reverse Time Dynamics

One of the core contributions is the derivation of the reverse time model, which is crucial for sampling from a generative model. By leveraging the continuous reparameterization and the properties of the newly defined process, the authors develop a coupled system of reverse-time SDEs. This system accounts for the correlation of processes involved in the generative modeling, thereby expanding traditional score-based models. The reverse-time dynamics differ notably from conventional models due to the infinite quadratic variation of FBM, introducing new challenges and opportunities for exploiting non-Markovian stochastic processes in generative tasks.

Experimental Insights

The authors put the theoretical constructs to the test using toy datasets, notably the Swiss roll and half-moon configurations. These experiments demonstrate the capacity of the GFDM to generate samples aligning closely with desired distributions, albeit with nuanced differences in distribution fidelity across varying Hurst indices and sampling methodologies. Notably, the paper explores sampling strategies that blend stochastic differential equations (SDE) with ordinary differential equations (ODE)-based approaches, such as the noisy ODE and guided SDE, yielding insights into the model's flexibility in handling correlation-driven dynamics.

Implications and Future Directions

This work propels the understanding of generative models into a new domain, harnessing the statistical properties of FBM. It provides a compelling case for applying non-Markovian processes in generative modeling tasks, offering an expanded toolkit for manipulating data distributions with fine control over path properties. The implications extend into machine learning, finance, and any field requiring sophisticated stochastic modeling.

Future explorations can extend to higher dimensions, generalization to space-time processes, and leveraging FBM's unique path and statistical characteristics for innovative applications. Additionally, the model's adaptability in managing correlation scales and variance shifts presents rich avenues for further research, particularly in developing more robust sampling strategies and exploring the full range of FBM capabilities.