Score-Based Generative Modeling through Stochastic Differential Equations (2011.13456v2)

Abstract: Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only on the time-dependent gradient field (\aka, score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

Score-Based Generative Modeling through Stochastic Differential Equations

Overview

The paper "Score-Based Generative Modeling through Stochastic Differential Equations" presents an innovative framework unifying score-based generative modeling and stochastic differential equations (SDEs). The authors, Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole, propose an SDE framework that incrementally transforms a complex data distribution into a simple, known prior by injecting noise and then reverses this process for data generation. The reverse process relies on the time-dependent gradient of the perturbed data distribution, which is estimated using neural networks.

Key Contributions

1. Unified Framework: The framework integrates previous score-based generative modeling (SMLD) and diffusion probabilistic modeling (DDPM) approaches through continuous SDEs. This unification offers new sampling procedures and capabilities, including a new predictor-corrector approach for error correction in discretized SDEs and an equivalent neural ODE for exact likelihood computation and enhanced sampling efficiency.
2. Theoretical and Practical Contributions:
   • Flexible Sampling and Likelihood Computation: Introduces general-purpose SDE solvers for sampling, predictor-corrector samplers combining numerical SDE solvers with score-based MCMC methods, and deterministic samplers based on probability flow ODEs, allowing flexible data manipulation and exact likelihood computation.
   • Controllable Generation: Enables modulation of the generation process by conditioning on previously unavailable information, supporting applications like class-conditional generation, image inpainting, and colorization without requiring model retraining.
   • Architectural Improvements: Through multiple architectural enhancements, the proposed approach achieves record-breaking performance for unconditional image generation, notably on CIFAR-10, with the potential for high-fidelity generation of large-scale images (e.g., $1024 \times 1024$ resolution).

Sampling and Likelihood Computation

The proposed methods for sampling include general-purpose numerical solvers, predictor-corrector samplers, and probability flow ODEs:

• General-Purpose SDE Solvers: These solvers discretize the reverse-time SDE to approximate sample generation trajectories, with reverse diffusion samplers outperforming ancestral sampling.
• Predictor-Corrector Samplers: Combining numerical SDE solvers with MCMC correctors, these samplers improve over predictor-only methods by correcting sample distributions at each step.
• Probability Flow ODEs: These enable exact likelihood computation via instantaneous change of variables and efficient sampling using black-box ODE solvers. Notably, they facilitate uniquely identifiable encodings, unlike typical invertible models, allowing for accurate latent space manipulation through interpolation and temperature scaling.

Empirical Results

The framework’s efficacy is substantiated through robust empirical results:

• Sample Quality and Likelihoods: Achieves state-of-the-art FID (2.20) and Inception Score (9.89) on CIFAR-10, and high-quality samples for large-scale images.
• Class-Conditional Generation: Demonstrated through training noise-conditional classifiers, enabling high-quality class-conditional sample generation.
• Image Imputation and Colorization: Addressed via conditional reverse-time SDEs, showing promising results in inpainting and colorization tasks.

Implications and Future Directions

The integration of score-based generative models with SDEs opens new avenues for probabilistic modeling, inference, and sample generation, providing a robust theoretical foundation and practical improvements in generative modeling. The framework's ability to handle controllable generation tasks without model retraining is particularly impactful, suggesting broad applicability across various domains.

Future developments may focus on further optimizing the hybrid predictors, exploring more efficient architectures for different SDE types, and merging the stable learning dynamics of score-based models with the rapid sampling capabilities of implicit models like GANs. Automating hyper-parameter tuning for the myriad sampling strategies introduced may also enhance applicability and performance.

In conclusion, the proposed SDE-based framework exemplifies a significant advancement in generative modeling, providing a versatile and theoretically sound approach that unifies and extends previous methods while setting new benchmarks for performance.