DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics (2310.13268v3)

Abstract: Diffusion probabilistic models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. However, they highly rely on specific parameterization during inference (such as noise/data prediction), which might not be the optimal choice. In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error of the ODE solution. Based on such formulation, we propose DPM-Solver-v3, a new fast ODE solver for DPMs by introducing several coefficients efficiently computed on the pretrained model, which we call empirical model statistics. We further incorporate multistep methods and a predictor-corrector framework, and propose some techniques for improving sample quality at small numbers of function evaluations (NFE) or large guidance scales. Experiments show that DPM-Solver-v3 achieves consistently better or comparable performance in both unconditional and conditional sampling with both pixel-space and latent-space DPMs, especially in 5$\sim$10 NFEs. We achieve FIDs of 12.21 (5 NFE), 2.51 (10 NFE) on unconditional CIFAR10, and MSE of 0.55 (5 NFE, 7.5 guidance scale) on Stable Diffusion, bringing a speed-up of 15%$\sim$30% compared to previous state-of-the-art training-free methods. Code is available at https://github.com/thu-ml/DPM-Solver-v3.

• The paper presents a novel ODE formulation that integrates empirical model statistics to reduce sampling inefficiencies in diffusion models.
• It employs a multistep predictor-corrector framework and pseudo-order strategies to enhance inference speed and reduce computational load.
• Experiments on CIFAR10 and MS-COCO show a 15-30% speed improvement and lower FID scores, highlighting the solver's practical impact.

An Analysis of "DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics"

The paper presents a new approach to improving the efficiency of sampling procedures within Diffusion Probabilistic Models (DPMs) through a novel Ordinary Differential Equation (ODE) formulation. These models have been pivotal in generating high-fidelity images but often encounter bottlenecks related to sampling inefficiency. The paper introduces DPM-Solver-v3, an ODE solver that integrates empirical model statistics (EMS) to enhance both unconditional and conditional sampling quality, particularly observable within 5 to 10 Number of Function Evaluations (NFEs).

Methodology and Technical Contributions

The authors propose a systematic investigation into model parameterization and ODE formulation, emphasizing the minimization of first-order discretization error. The paper introduces EMS—coefficients computed on pretrained models—that facilitate this reduction in error. The research integrates elements of Rosenbrock-type exponential integrators and first-order discretization analysis to determine the optimal parameterization strategy during inference.

The EMS is analytically derived and incorporates:

• Three types of coefficients: L, S, and B, which adjust the model's semi-linear structure in solving diffusion ODEs.
• A generalized parameterization (gθ) that extends beyond traditional noise/data prediction.

The paper further proposes a multistep predictor-corrector framework and practical techniques like pseudo-order solvers and half-corrector strategies for enhancing sample quality, especially under constraints of small NFE or large guidance scales.

Experimental Results and Implications

DPM-Solver-v3 demonstrates superior or comparable performance across various datasets and configurations. Key results indicate a 15-30% improvement in speed, with notable advancements in reducing the Fréchet Inception Distance (FID) in image generation tasks:

• On CIFAR10 with ScoreSDE, the solver achieves an FID of 12.76 (5 NFE) and 2.71 (20 NFE).
• For large-scale datasets and models like Stable-Diffusion with MS-COCO2014 prompts, the solver attains convergence more rapidly, as evaluated by the Mean Squared Error (MSE) in latent space.

Beyond theoretical analysis, practical impacts are tangible in areas like real-time generation where reduced NFEs directly correlate with reduced computational load and costs, which is vital for real-time applications in AIGC and other domains.

Future Directions and Considerations

The paper posits an advancement in the parameterization strategy of diffusion ODEs that could extend to diverse generative models. However, the scalability to real-time applications remains constrained by the innate characteristics of training-free methods. Further research might explore adaptive scheduling strategies that could complement the ODE solver framework, potentially minimizing NFEs even further while maintaining high sample fidelity.

In terms of broader impact, while the improved efficiency could positively affect high-demand image synthesis applications, ethical concerns regarding the generation of highly realistic images persist. As diffusion models become faster and more efficient, continued exploration of their ethics and potential limitations will be crucial.

In summary, "DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics" offers valuable insights and methods for addressing sampling inefficiencies in diffusion models, with practical and theoretical advancements shaping future AI generations.