---
title: Third-Order DPM-Solver++ Overview
url: https://www.emergentmind.com/topics/third-order-dpm-solver
type: topic
---

# Third-Order DPM-Solver++ Overview

Third-order DPM-Solver++ refers to the (hypothetical) extension of the DPM-Solver++ algorithm, a class of ODE solvers for efficient sampling in diffusion probabilistic models (DPMs), to third-order global accuracy. While DPM-Solver++ is designed to accelerate both unguided and guided sampling from DPMs by leveraging high-order numerical integration, the DPM-Solver++ framework as described in the foundational literature only explicitly establishes and analyzes solvers up to second order (k=2), with the authors cautioning that solvers of order k ≥ 3 remain an open research problem, particularly due to observed instabilities at high guidance scales [2211.01095]. In contrast, the third-order high-accuracy adaptive solver is fully detailed for the original DPM-Solver family [2206.00927], encompassing both theoretical derivation and implementation aspects.

## 1. Theoretical Foundation and ODE Formulation

DPM-Solver++ operates on the continuous-time denoising ODE associated with diffusion probabilistic models. This ODE is parameterized either by a noise prediction model $\epsilon_\theta(x_t, t)$ or a data prediction model $x_{0,\theta}(x_t, t)$:

- **Noise prediction parameterization:**
  $$
  \frac{dx_t}{dt} = f(t)x_t + \frac{g^2(t)}{2\sigma_t}\epsilon_\theta(x_t, t)
  $$
- **Data prediction parameterization:**
  $$
  \frac{dx_t}{dt} = \left(f(t) + \frac{g^2(t)}{2\sigma_t^2}\right)x_t - \frac{\alpha_t g^2(t)}{2\sigma_t^2}x_{0,\theta}(x_t, t)
  $$

Here, $t$ is the diffusion time, $f(t)$ and $g^2(t)$ relate to the variance-preserving (VP) SDE parameters, and $\alpha_t^2 / \sigma_t^2$ (SNR) is monotonically decreasing [2211.01095]. The ODE admits an exact integral solution that forms the basis for constructing high-order numerical solvers [2206.00927].

## 2. High-Order ODE Solvers: DPM-Solver and DPM-Solver++

The DPM-Solver framework provides a general recipe for constructing explicit high-order, stiffly accurate exponential integrators for the denoising ODE. For order-$k$, this involves a Taylor expansion of the neural network prediction ($\hat\epsilon$) up to $k-1$ derivatives in a log-SNR parameterization ($\lambda$):

- **General update:** 
  $$
  x_{t_{i}} = \frac{\alpha_{i}}{\alpha_{i-1}} x_{i-1} - \sigma_{i} \sum_{j=1}^k h^j \varphi_j(h)\frac{\hat\epsilon^{(j-1)}_0}{(j-1)!}
  $$
where $\varphi_j(h)$ are exponential integrator kernels, $h = \lambda_i - \lambda_{i-1}$, and $\hat\epsilon^{(j)}_0$ denotes the $j$-th derivative at $\lambda_{i-1}$ [2206.00927].

DPM-Solver++ adapts these ideas, employing the data-prediction parameterization, dynamic thresholding (clipping $x_{0,\theta}$ to bounded ranges), and multistep formulations to address instabilities observed in guided sampling [2211.01095]. However, the literature provides explicit derivations only for $k=2$ (second order), not $k=3$.

## 3. Third-Order Methods: DPM-Solver vs. DPM-Solver++

The third-order DPM-Solver (DPM-Solver-3) is fully developed in [2206.00927], offering explicit update formulas, rigorous global accuracy proofs, and practical pseudocode. The solver uses intermediate sub-steps at $r_1=1/3$ and $r_2=2/3$ of each time interval to avoid explicit computation of neural network derivatives, and these sub-steps are analytically constructed to satisfy all order conditions. The essential steps are summarized in the following table:

| Algorithmic Component      | DPM-Solver-3                         | DPM-Solver++ (k=3)                |
|---------------------------|--------------------------------------|------------------------------------|
| Update formula            | Explicit, with two intermediate sub-steps | Not present in literature         |
| Pseudocode                | Fully specified                       | Not specified for $k=3$           |
| Global accuracy proof     | $O(h_\mathrm{max}^3)$                 | Not available for $k=3$            |
| Guidance/thresholding     | Works for basic and moderate guidance | Instability at large guidance, not pursued [2211.01095] |

A plausible implication is that, although the generalized Taylor-expansion template for arbitrary $k$ is shown for DPM-Solver++ [2211.01095, Eq. (7)], explicit construction of a third-order method remains prospective, and there are no published algorithms, error analyses, or empirical studies for DPM-Solver++ at $k=3$.

## 4. Stability and Instability for High-Order Solvers

Empirical evidence in [2211.01095] demonstrates that high-order (k ≥ 3) solvers—whether from DPM-Solver, DEIS, or other families—exhibit severe numerical instability when applied to guided sampling with large classifier or classifier-free guidance scales (e.g., $s \gtrsim 4.0$). In such regimes, the solvers can produce divergence or blows up in practice, especially when attempting aggressive NFE (number of function evaluations) reduction to $O(10)$. This is attributed to the Lipschitz constants and local nonlinearity being amplified by the guidance signal and cannot be simply remedied via thresholding or naive step size reduction. Thus, DPM-Solver++ restricts its practical implementations and theoretical development to $k \leq 2$ [2211.01095].

## 5. Adaptive Step Size and “++” Enhancements

Adaptive step size control (“++” enhancement) is constructed by embedding lower- and higher-order solvers (e.g., orders 2 and 3) in a stepwise local error estimate. DPM-Solver-23 performs both second- and third-order updates, forming an error estimate $E = \Vert \Delta x_2 - \Delta x_3 \Vert / \text{tol}$ and updating $h \to h \cdot \min(5, \max(0.2, E^{-1/3}))$. Steps are accepted when $E \leq 1$; otherwise, $h$ is reduced and retried [2206.00927, Appendix D.2]. This method yields 20–50% reductions in total NFE while maintaining full third-order accuracy. DPM-Solver++ in its current published form does not extend this adaptive machinery to third order due to the aforementioned stability problems for guidance and lack of an explicit $k=3$ scheme [2211.01095].

## 6. Practical Implications and Implementation Recommendations

- DPM-Solver++ as scientifically established supports single- and multistep methods up to $k=2$. For classifier-free guidance scales $s \gtrsim 7.5$, multistep second-order integration is recommended as it reduces effective step size per empirical findings [2211.01095].
- Uniform $t$ (time) scheduling outperforms uniform $\lambda$ (log-SNR) based grids in practical DPM-Solver++ usage; when thresholding is used, data-prediction models ($x_{0,\theta}$) should be preferred.
- For the original DPM-Solver framework, both fixed and adaptive third-order integration works robustly in basic DPM sampling, but guidance scenarios at high scales are not addressed.
- There is no published algorithm, pseudocode, or error analysis for DPM-Solver++ at $k=3$.

## 7. Comparison with Prior High-Order Solvers and Open Directions

Comparative benchmarking in [2211.01095] establishes that second-order DPM-Solver++ consistently outperforms first-order methods (e.g., DDIM) at low NFE regimes ($\lesssim 20$ steps) and matches or exceeds the sample quality of PNDM and similar high-order solvers, particularly when used with multistep control and dynamic thresholding. In contrast, third-order methods from prior work (DPM-Solver, DEIS) underperform at large guidance amplitudes, sometimes trailing behind even the basic first-order samplers. The extension of DPM-Solver++ to third-order and beyond, especially with empirical stability for guided DPM sampling, is explicitly left open for future research [2211.01095].

Source: https://www.emergentmind.com/topics/third-order-dpm-solver