Multistep Distillation of Diffusion Models

• Multistep distillation of diffusion models is a process that condenses hundreds of denoising steps into a few efficient operations via operator merging.
• It employs dynamic programming to determine optimal merge schedules that balance speed and fidelity based on data covariance properties.
• Empirical results demonstrate that sequential BOOT, one-shot, or hybrid strategies can reduce reconstruction error by up to 30% in targeted regimes.

Multistep distillation of diffusion models encompasses a range of techniques for accelerating the sampling process of diffusion generative models by replacing a many-step denoising trajectory with a procedure involving far fewer steps, often as few as a single step. This article synthesizes technical principles, theoretical frameworks, algorithmic forms, and key empirical results from recent literature, with emphasis on the operator-merging perspective introduced in Gao & Li (2024) and connections to the broader methodological landscape.

1. Conceptual Foundations: Diffusion Sampling as Operator Trajectories

At the core of diffusion generative modeling is a learned multi-step reverse process that stochastically removes noise from a sample, reconstructing clean data from a noise prior. Each denoising step in the discretized reverse process can be regarded as application of a parameterized map (in the linear regime, a coordinate-wise linear operator) to the latent variable. The sequence of maps, parameterized by the noise schedule and data covariance, collectively define the teacher’s sampling trajectory. Mathematically, with $x_0\sim\mathcal N(0, \Lambda)$, the forward process at discrete time $t$ operates as

$$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$

with $\alpha_t^2 + \sigma_t^2 = 1$. The optimal denoiser under $\ell_2$ loss is itself a linear operator:

$$\hat x_0(z_t, t) = \mathbb E[x_0|z_t] = \alpha_t \Lambda (\alpha_t^2 \Lambda + \sigma_t^2 I)^{-1} z_t.$$

The reverse DDIM-style step is a diagonal linear projection/rescaling, with coordinate update

$$(z_{t-1})_i = (A_t^*)_i (z_t)_i, \quad (A_t^*)_i = \frac{\alpha_{t-1} \alpha_t \lambda_i + \sigma_{t-1} \sigma_t}{\alpha_t^2 \lambda_i + \sigma_t^2}.$$

A full teacher trajectory over $K$ steps is thus a product of these diagonal operators, $(\mathcal T_K)_i = \prod_j (A_j^*)_i$ (Gao et al., 21 May 2025).

2. Multistep Distillation via Operator Merging

Multistep distillation seeks to approximate the teacher trajectory—typically spanning hundreds to thousands of steps—by training a student model whose sampling consists of substantially fewer steps, possibly just one. In the operator-merging perspective, this corresponds to merging contiguous blocks of the teacher's operators into coarse-step student operators. The merging process can be formulated recursively: merging $[t_1+1, ..., t_2]$ yields for each coordinate

$t$0

where $t$1, and $t$2. This convex combination captures both the contraction arising from discretization and the finite optimization budget for the student. Geometrically, every merge stage corresponds to a weighted blend between the full sequence of projections (student target) and the last step projection (teacher's last operator), with signal loss quantified as shrinkage of the effective projection norm (Gao et al., 21 May 2025).

3. Dynamic Programming for Optimal Merge Planning

Determining the merge schedule with minimal signal loss is a discrete optimization problem over possible partitions of the teacher's step sequence. This is efficiently solved by dynamic programming (DP) on the covariances, seeking to minimize the squared $t$3-distance between the ideal composite operator $t$4 and any candidate merge plan:

$t$5

For every interval $t$6, DP computes the best merge by recursively considering all splits and direct merges. The approach leverages coordinate-wise convexity, ensuring global optimality under the surrogate loss. The total DP runtime is $t$7 (Gao et al., 21 May 2025).

4. Data Regimes and Phase Transition in Merge Strategies

The optimal merge plan is sharply determined by the data covariance structure, specifically by the dominant eigenvalues $t$8 of $t$9. Two asymptotic regimes arise in the scalar case ($$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$0):

• Low-variance regime ($$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$1): All $$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$2 and the optimal DP solution is sequential BOOT—merging one step at a time, i.e., minimal step size reduction at each merge.
• High-variance regime ($$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$3): For $$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$4, $$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$5 (amplification before shrinkage). Here, the optimal plan is vanilla trajectory distillation: merge all teacher steps in a single one-shot student.
• Intermediate regime ($$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$6): The DP discovers nontrivial hybrid merge plans mixing the behaviors above (Gao et al., 21 May 2025).

A critical threshold $$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$7, with $$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$8, precisely demarcates the “phase transition” between these regimes. Empirically, as $$z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),$$9 moves from $\alpha_t^2 + \sigma_t^2 = 1$0 to $\alpha_t^2 + \sigma_t^2 = 1$1, the error advantage shifts correspondingly from sequential BOOT to vanilla, with DP merges outperforming both for $\alpha_t^2 + \sigma_t^2 = 1$2 in $\alpha_t^2 + \sigma_t^2 = 1$3.

5. Empirical and Practical Implications

Empirical results on synthetic Gaussian data set the error ordering for different strategies:

• $\alpha_t^2 + \sigma_t^2 = 1$4: sequential BOOT yields lowest $\alpha_t^2 + \sigma_t^2 = 1$5 error.
• $\alpha_t^2 + \sigma_t^2 = 1$6: vanilla one-shot distillation dominates.
• Intermediate $\alpha_t^2 + \sigma_t^2 = 1$7: dynamically planned merges via DP can lower errors by $\alpha_t^2 + \sigma_t^2 = 1$8–$\alpha_t^2 + \sigma_t^2 = 1$9 relative to either BOOT or vanilla.

Experiments on real latent-space datasets show that diagonal covariances $\ell_2$0 favor sequential BOOT, which delivers better reconstructions and lower pixel-space $\ell_2$1 error throughout training. The operator-merging framework thus delivers actionable recommendations:

1. For data/latents with $\ell_2$2, use sequential BOOT—merge steps one-by-one.
2. For $\ell_2$3, use vanilla one-shot distillation—merge all at once.
3. For ambiguous or intermediate $\ell_2$4, run the DP planner to find the optimal schedule (Gao et al., 21 May 2025).

6. Theoretical Significance and Connections

The operator-merging interpretation of multistep distillation formalizes, in the linear regime, why and how different trade-offs between speed (number of student steps) and generative fidelity arise. Each step of a DDIM teacher is a diagonal projection/rescaling, and the only feasible student is a convex combination thereof, subject to signal loss due to finite merging. Minimizing the resultant $\ell_2$5 error via DP not only explains the empirical findings of prior methods—including sequential reduction (BOOT), one-shot distillation, and progressive halving schemes—but reveals the presence of a nontrivial “phase diagram” controlled by the data’s covariance spectrum.

This framework also clarifies why standard progressive distillation (Salimans et al., 2022), consistency models (Heek et al., 2024), and trajectory-matching approaches perform variably across tasks: the regime dictated by data variance directs optimal schedule design for multistep-to-few-step compression. The approach is agnostic to network class or loss, highlighting a fundamental property of the denoising trajectory rather than any architecture-specific effect (Gao et al., 21 May 2025).

7. Outlook and Open Directions

Operator-merging analysis currently applies primarily to linear Gaussian approximations, but suggests extensibility to more general (e.g., non-Gaussian, nonlinear) generative processes by considering the action of nonlinear denoising operators on higher moments or local projections. Automatic merge-schedule planners, based on empirically estimated data covariance, can be deployed to adapt step-reduction strategies for arbitrary datasets or conditional distributions. Extensions to high-dimensional structured output spaces and richer noise schedules represent concrete directions for future work.

The operator-merging view of multistep diffusion model distillation thus provides a unified understanding of the available distillation regimes, enables principled speed/fidelity trade-offs, and offers a sound basis for algorithmic innovation (Gao et al., 21 May 2025).