First-Order VE ER-SDE-Solver

• The VE ER-SDE-Solver-1 offers a training-free, first-order approach that interpolates between ODE and SDE sampling through a tunable reverse-time noise scale.
• It employs an Euler-style update derived via a one-step Taylor expansion, achieving a global order of accuracy O(Δσ) with a single network evaluation per step.
• Empirical results on ImageNet indicate competitive FID scores, demonstrating a practical balance between sampling speed, diversity, and generation quality.

A first-order VE ER-SDE-Solver (often denoted as "VE ER-SDE-Solver-1") is a stochastic differential equation (SDE) solver for diffusion models, derived within the framework of Extended Reverse-Time SDEs. This method generalizes and interpolates between ordinary differential equation (ODE) and SDE-based diffusion samplers by introducing a tunable reverse-process noise scale. Specifically, it targets the variance-exploding (VE) case, delivering a computationally lightweight, training-free algorithm that attains state-of-the-art performance among first-order stochastic solvers in diffusion-based generative modeling (Cui et al., 2023).

1. Mathematical Formulation

The Variance-Exploding Extended Reverse-Time SDE (VE ER SDE) is defined by setting $f(t)=0$ and $g(t)=\sqrt{d\,\sigma^2(t)/dt}$, where $\sigma(t)$ describes the forward-process noise schedule. An additional reverse-process noise scale $h(t)$, with $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$, is introduced. The ER SDE is then: $$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$ where $\overline{w}_t$ is a reverse-time Wiener process, $x_\theta(\cdot, \cdot)$ is the pretrained data model, and $\xi(\sigma), \varphi(\sigma)$ control the reverse-process scale. After time-reparameterization ($\sigma \leftrightarrow t$), the SDE becomes

$g(t)=\sqrt{d\,\sigma^2(t)/dt}$0

The function $g(t)=\sqrt{d\,\sigma^2(t)/dt}$1 is strictly increasing and satisfies $g(t)=\sqrt{d\,\sigma^2(t)/dt}$2. This parameterization allows interpolating between deterministic and stochastic sampling regimes (Cui et al., 2023).

2. First-Order Euler-Style Update Rule

An explicit first-order ("Euler-style") scheme is derived via a one-step Taylor expansion. The formal exact solution (Proposition 3.2) is: $g(t)=\sqrt{d\,\sigma^2(t)/dt}$3 where $g(t)=\sqrt{d\,\sigma^2(t)/dt}$4. To make this practical, $g(t)=\sqrt{d\,\sigma^2(t)/dt}$5 in the integral is replaced with $g(t)=\sqrt{d\,\sigma^2(t)/dt}$6 (left-endpoint evaluation). The resulting update is: $g(t)=\sqrt{d\,\sigma^2(t)/dt}$7 where $g(t)=\sqrt{d\,\sigma^2(t)/dt}$8.

3. Algorithmic Execution

The VE ER-SDE-Solver-1 operates in discrete time over a user-supplied time grid $g(t)=\sqrt{d\,\sigma^2(t)/dt}$9 with precomputed noise schedules and reverse-process scaling arrays. The requisite steps are summarized in the following table:

Step	Operation	Notes
Initialization	Set $\sigma(t)$0	$\sigma(t)$1: standard normal noise
For each $\sigma(t)$2	$\sigma(t)$3
	$\sigma(t)$4	Gaussian noise
	$\sigma(t)$5	One forward network eval
	$\sigma(t)$6	Euler-style update
Return	$\sigma(t)$7	Output sample

This scheme induces a one-step truncation error of $\sigma(t)$8, and global order of accuracy $\sigma(t)$9, classifying it as a first-order method (Cui et al., 2023).

4. Computational Considerations

Each step in VE ER-SDE-Solver-1 requires a single evaluation of the pretrained data model $h(t)$0, generation of one Gaussian noise sample, and $h(t)$1 vector-scalar arithmetic (with $h(t)$2 the data dimension). Its computational footprint matches that of ancestral SDE sampling and single-evaluation ODE solvers. It is more lightweight compared to high-order solvers, which involve derivative evaluations or multi-stage integration.

Runtime is similar to the EDM-Stochastic or DDIM ($h(t)$3) samplers, and marginally slower than deterministic ODE-only solvers due to the need for a Gaussian draw. The method is suitable for batch or parallel settings, leveraging modern accelerators effectively (Cui et al., 2023).

5. Empirical Performance Metrics

Empirical evaluation on the Guided-diffusion ImageNet $h(t)$4 model (linear noise schedule) establishes the competitiveness of VE ER-SDE-Solver-1. As measured by Fréchet Inception Distance (FID) at $h(t)$5 (network function evaluations):

• $h(t)$6: FID $h(t)$7
• $h(t)$8: FID $h(t)$9
• $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$0: FID $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$1
• $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$2: FID $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$3

At $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$4:

• DDIM ($h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$5): FID $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$6
• SDE-DPM-Solver++(2M): FID $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$7
• VE ER-SDE-Solver-1: FID $h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$8--$h^2(t)=\xi(t)\,(d\,\sigma^2(t)/dt)/(2\sigma(t))$9

Compared to higher-order VE ER-SDE-Solver-3, which achieves FID $$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$0 at $$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$1, VE ER-SDE-Solver-1 situates performance midway between DDIM and advanced stochastic solvers. This suggests that the first-order solver offers substantial improvements over previous methods while retaining full stochasticity (Cui et al., 2023).

6. Theoretical and Practical Insights

A tunable scaling function $$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$2 allows VE ER-SDE-Solver-1 to interpolate continuously between deterministic ODE sampling ($$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$3) and high-variance SDE sampling ($$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$4). By choosing an optimal $$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$5, the discretization error term $$d x_t = \left[ \frac{d\sigma^2(t)/dt + h^2(t)}{2\sigma^2(t)} (x_t - x_\theta(x_t, t)) \right]dt + h(t)\, d\overline{w}_t,$$6 can be minimized, enhancing both quality and speed. The injected noise term corrects for bias accumulated early in simulation, yielding improved sample diversity as a "Langevin-style" correction.

Advantages include:

• Substantial narrowing of the quality gap to ODE-based methods, even with single network function evaluations per step.
• Maintenance of the benefits of stochastic sampling (greater diversity).
• Outperforming all previously published first-order SDE samplers.

A plausible implication is that VE ER-SDE-Solver-1 constitutes a practical, "turn-key," training-free sampler, suitable for diverse application settings requiring a balance of sampling speed and generative quality (Cui et al., 2023).

7. Applications and Significance

The first-order VE ER-SDE-Solver plays a central role in the toolkit of modern diffusion-based generative frameworks, enabling efficient sampling at a level of quality previously associated with higher-order methods or increased function evaluations. Its lightweight computational profile, coupled with tunable stochasticity, makes it well-suited for scenarios demanding high-throughput sampling or ensemble generation. The methodology unifies ODE and SDE approaches, allowing flexible traversal of the speed-diversity-quality tradeoff landscape in generative modeling (Cui et al., 2023).