Third-Order DPM-Solver++ Overview

• Third-order DPM-Solver++ is a conceptual extension of the DPM-Solver++ framework, pushing beyond the second-order limit to achieve higher global accuracy in ODE solvers for diffusion models.
• The methodology builds on high-order Taylor expansions and adaptive step size control, similar to DPM-Solver-23, to potentially reduce function evaluations while maintaining theoretical accuracy.
• Practical implementations recommend second-order methods for guided sampling due to observed instabilities at high guidance scales, leaving third-order derivatives as a promising open research problem.

Third-order DPM-Solver++ refers to the (hypothetical) extension of the DPM-Solver++ algorithm, a class of ODE solvers for efficient sampling in diffusion probabilistic models (DPMs), to third-order global accuracy. While DPM-Solver++ is designed to accelerate both unguided and guided sampling from DPMs by leveraging high-order numerical integration, the DPM-Solver++ framework as described in the foundational literature only explicitly establishes and analyzes solvers up to second order (k=2), with the authors cautioning that solvers of order k ≥ 3 remain an open research problem, particularly due to observed instabilities at high guidance scales (Lu et al., 2022). In contrast, the third-order high-accuracy adaptive solver is fully detailed for the original DPM-Solver family (Lu et al., 2022), encompassing both theoretical derivation and implementation aspects.

1. Theoretical Foundation and ODE Formulation

DPM-Solver++ operates on the continuous-time denoising ODE associated with diffusion probabilistic models. This ODE is parameterized either by a noise prediction model $\epsilon_\theta(x_t, t)$ or a data prediction model $x_{0,\theta}(x_t, t)$:

• Noise prediction parameterization:

$$\frac{dx_t}{dt} = f(t)x_t + \frac{g^2(t)}{2\sigma_t}\epsilon_\theta(x_t, t)$$

• Data prediction parameterization:

$$\frac{dx_t}{dt} = \left(f(t) + \frac{g^2(t)}{2\sigma_t^2}\right)x_t - \frac{\alpha_t g^2(t)}{2\sigma_t^2}x_{0,\theta}(x_t, t)$$

Here, $t$ is the diffusion time, $f(t)$ and $g^2(t)$ relate to the variance-preserving (VP) SDE parameters, and $\alpha_t^2 / \sigma_t^2$ (SNR) is monotonically decreasing (Lu et al., 2022). The ODE admits an exact integral solution that forms the basis for constructing high-order numerical solvers (Lu et al., 2022).

2. High-Order ODE Solvers: DPM-Solver and DPM-Solver++

The DPM-Solver framework provides a general recipe for constructing explicit high-order, stiffly accurate exponential integrators for the denoising ODE. For order-$k$, this involves a Taylor expansion of the neural network prediction ($\hat\epsilon$) up to $k-1$ derivatives in a log-SNR parameterization ($\lambda$):

• General update:

$$x_{t_{i}} = \frac{\alpha_{i}}{\alpha_{i-1}} x_{i-1} - \sigma_{i} \sum_{j=1}^k h^j \varphi_j(h)\frac{\hat\epsilon^{(j-1)}_0}{(j-1)!}$$

where $\varphi_j(h)$ are exponential integrator kernels, $h = \lambda_i - \lambda_{i-1}$, and $\hat\epsilon^{(j)}_0$ denotes the $j$-th derivative at $\lambda_{i-1}$ (Lu et al., 2022).

DPM-Solver++ adapts these ideas, employing the data-prediction parameterization, dynamic thresholding (clipping $x_{0,\theta}$ to bounded ranges), and multistep formulations to address instabilities observed in guided sampling (Lu et al., 2022). However, the literature provides explicit derivations only for $k=2$ (second order), not $k=3$.

3. Third-Order Methods: DPM-Solver vs. DPM-Solver++

The third-order DPM-Solver (DPM-Solver-3) is fully developed in (Lu et al., 2022), offering explicit update formulas, rigorous global accuracy proofs, and practical pseudocode. The solver uses intermediate sub-steps at $r_1=1/3$ and $r_2=2/3$ of each time interval to avoid explicit computation of neural network derivatives, and these sub-steps are analytically constructed to satisfy all order conditions. The essential steps are summarized in the following table:

Algorithmic Component	DPM-Solver-3	DPM-Solver++ (k=3)
Update formula	Explicit, with two intermediate sub-steps	Not present in literature
Pseudocode	Fully specified	Not specified for $k=3$
Global accuracy proof	$O(h_\mathrm{max}^3)$	Not available for $k=3$
Guidance/thresholding	Works for basic and moderate guidance	Instability at large guidance, not pursued (Lu et al., 2022)

A plausible implication is that, although the generalized Taylor-expansion template for arbitrary $k$ is shown for DPM-Solver++ [(Lu et al., 2022), Eq. (7)], explicit construction of a third-order method remains prospective, and there are no published algorithms, error analyses, or empirical studies for DPM-Solver++ at $k=3$.

4. Stability and Instability for High-Order Solvers

Empirical evidence in (Lu et al., 2022) demonstrates that high-order (k ≥ 3) solvers—whether from DPM-Solver, DEIS, or other families—exhibit severe numerical instability when applied to guided sampling with large classifier or classifier-free guidance scales (e.g., $s \gtrsim 4.0$). In such regimes, the solvers can produce divergence or blows up in practice, especially when attempting aggressive NFE (number of function evaluations) reduction to $O(10)$. This is attributed to the Lipschitz constants and local nonlinearity being amplified by the guidance signal and cannot be simply remedied via thresholding or naive step size reduction. Thus, DPM-Solver++ restricts its practical implementations and theoretical development to $k \leq 2$ (Lu et al., 2022).

5. Adaptive Step Size and “++” Enhancements

Adaptive step size control (“++” enhancement) is constructed by embedding lower- and higher-order solvers (e.g., orders 2 and 3) in a stepwise local error estimate. DPM-Solver-23 performs both second- and third-order updates, forming an error estimate $$E = \Vert \Delta x_2 - \Delta x_3 \Vert / \text{tol}$$ and updating $h \to h \cdot \min(5, \max(0.2, E^{-1/3}))$. Steps are accepted when $E \leq 1$; otherwise, $h$ is reduced and retried [(Lu et al., 2022), Appendix D.2]. This method yields 20–50% reductions in total NFE while maintaining full third-order accuracy. DPM-Solver++ in its current published form does not extend this adaptive machinery to third order due to the aforementioned stability problems for guidance and lack of an explicit $k=3$ scheme (Lu et al., 2022).

6. Practical Implications and Implementation Recommendations

• DPM-Solver++ as scientifically established supports single- and multistep methods up to $k=2$. For classifier-free guidance scales $s \gtrsim 7.5$, multistep second-order integration is recommended as it reduces effective step size per empirical findings (Lu et al., 2022).
• Uniform $t$ (time) scheduling outperforms uniform $\lambda$ (log-SNR) based grids in practical DPM-Solver++ usage; when thresholding is used, data-prediction models ($x_{0,\theta}$) should be preferred.
• For the original DPM-Solver framework, both fixed and adaptive third-order integration works robustly in basic DPM sampling, but guidance scenarios at high scales are not addressed.
• There is no published algorithm, pseudocode, or error analysis for DPM-Solver++ at $k=3$.

7. Comparison with Prior High-Order Solvers and Open Directions

Comparative benchmarking in (Lu et al., 2022) establishes that second-order DPM-Solver++ consistently outperforms first-order methods (e.g., DDIM) at low NFE regimes ($\lesssim 20$ steps) and matches or exceeds the sample quality of PNDM and similar high-order solvers, particularly when used with multistep control and dynamic thresholding. In contrast, third-order methods from prior work (DPM-Solver, DEIS) underperform at large guidance amplitudes, sometimes trailing behind even the basic first-order samplers. The extension of DPM-Solver++ to third-order and beyond, especially with empirical stability for guided DPM sampling, is explicitly left open for future research (Lu et al., 2022).