---
title: Multistep Distillation of Diffusion Models
url: https://www.emergentmind.com/topics/multistep-distillation-of-diffusion-models
type: topic
---

# Multistep Distillation of Diffusion Models

Multistep distillation of diffusion models encompasses a range of techniques for accelerating the sampling process of diffusion generative models by replacing a many-step denoising trajectory with a procedure involving far fewer steps, often as few as a single step. This article synthesizes technical principles, theoretical frameworks, algorithmic forms, and key empirical results from recent literature, with emphasis on the operator-merging perspective introduced in Gao & Li (2024) and connections to the broader methodological landscape.

## 1. Conceptual Foundations: Diffusion Sampling as Operator Trajectories

At the core of diffusion generative modeling is a learned multi-step reverse process that stochastically removes noise from a sample, reconstructing clean data from a noise prior. Each denoising step in the discretized reverse process can be regarded as application of a parameterized map (in the linear regime, a coordinate-wise linear operator) to the latent variable. The sequence of maps, parameterized by the noise schedule and data covariance, collectively define the teacher’s sampling trajectory. Mathematically, with $x_0\sim\mathcal N(0, \Lambda)$, the forward process at discrete time $t$ operates as
$$
z_t = \alpha_t x_0 + \sigma_t \epsilon,\quad \epsilon \sim \mathcal N(0, I),
$$
with $\alpha_t^2 + \sigma_t^2 = 1$. The optimal denoiser under $\ell_2$ loss is itself a linear operator:
$$
\hat x_0(z_t, t) = \mathbb E[x_0|z_t] = \alpha_t \Lambda (\alpha_t^2 \Lambda + \sigma_t^2 I)^{-1} z_t.
$$
The reverse DDIM-style step is a diagonal linear projection/rescaling, with coordinate update
$$
(z_{t-1})_i = (A_t^*)_i (z_t)_i, \quad (A_t^*)_i = \frac{\alpha_{t-1} \alpha_t \lambda_i + \sigma_{t-1} \sigma_t}{\alpha_t^2 \lambda_i + \sigma_t^2}.
$$
A full teacher trajectory over $K$ steps is thus a product of these diagonal operators, $(\mathcal T_K)_i = \prod_j (A_j^*)_i$ [2505.16024].

## 2. Multistep Distillation via Operator Merging

Multistep distillation seeks to approximate the teacher trajectory—typically spanning hundreds to thousands of steps—by training a student model whose sampling consists of substantially fewer steps, possibly just one. In the operator-merging perspective, this corresponds to merging contiguous blocks of the teacher's operators into coarse-step student operators. The merging process can be formulated recursively: merging $[t_1+1, ..., t_2]$ yields for each coordinate
$$
(A_{t_1+1 : t_2})_i = (1 - \gamma_{t_2}^{(i)}) \prod_{t=t_1+1}^{t_2} (A_t^*)_i + \gamma_{t_2}^{(i)} (A_{t_2}^*)_i,
$$
where $\gamma_{t_2}^{(i)} = e^{-2 s \|v_{t_2}^{(i)}\|^2}$, and $v_t^{(i)} = (\alpha_t \sqrt{\lambda_i}, \sigma_t)^\top$. This convex combination captures both the contraction arising from discretization and the finite optimization budget for the student. Geometrically, every merge stage corresponds to a weighted blend between the full sequence of projections (student target) and the last step projection (teacher's last operator), with signal loss quantified as shrinkage of the effective projection norm [2505.16024].

## 3. Dynamic Programming for Optimal Merge Planning

Determining the merge schedule with minimal signal loss is a discrete optimization problem over possible partitions of the teacher's step sequence. This is efficiently solved by dynamic programming (DP) on the covariances, seeking to minimize the squared $W_2$-distance between the ideal composite operator $\widetilde{\mathcal T}_T$ and any candidate merge plan:
$$
\sum_i \left((\widetilde{\mathcal T}_T)_i - (A_{1:T})_i\right)^2.
$$
For every interval $[t_1+1, t_2]$, DP computes the best merge by recursively considering all splits and direct merges. The approach leverages coordinate-wise convexity, ensuring global optimality under the surrogate loss. The total DP runtime is $O(d T^2)$ [2505.16024].

## 4. Data Regimes and Phase Transition in Merge Strategies

The optimal merge plan is sharply determined by the data covariance structure, specifically by the dominant eigenvalues $\lambda_i$ of $\Lambda$. Two asymptotic regimes arise in the scalar case ($\lambda=\Lambda$):

- **Low-variance regime ($\lambda\leq 1$):** All $(A_t^*)_i \leq 1$ and the optimal DP solution is sequential BOOT—merging one step at a time, i.e., minimal step size reduction at each merge.
- **High-variance regime ($\lambda \gg 1$):** For $t<T$, $(A_t^*)_i > 1$ (amplification before shrinkage). Here, the optimal plan is vanilla trajectory distillation: merge all teacher steps in a single one-shot student.
- **Intermediate regime ($\lambda \approx 1$):** The DP discovers nontrivial hybrid merge plans mixing the behaviors above [2505.16024].

A critical threshold $\lambda_0^*=\max_{t<T} \lambda_0(t)$, with $\lambda_0(t)=\tfrac{\sigma_t(\sigma_t-\sigma_{t-1})}{\alpha_t(\alpha_{t-1}-\alpha_t)}$, precisely demarcates the “phase transition” between these regimes. Empirically, as $\lambda$ moves from $0.5$ to $2.5$, the error advantage shifts correspondingly from sequential BOOT to vanilla, with DP merges outperforming both for $\lambda$ in $[1,2]$.

## 5. Empirical and Practical Implications

Empirical results on synthetic Gaussian data set the error ordering for different strategies:

- $\lambda \leq 1$: sequential BOOT yields lowest $W_2$ error.
- $\lambda \gg 1$: vanilla one-shot distillation dominates.
- Intermediate $\lambda$: dynamically planned merges via DP can lower errors by $10\%$–$30\%$ relative to either BOOT or vanilla.

Experiments on real latent-space datasets show that diagonal covariances $\lambda_i \lesssim 1$ favor sequential BOOT, which delivers better reconstructions and lower pixel-space $L_2$ error throughout training. The operator-merging framework thus delivers actionable recommendations:

1. For data/latents with $\lambda_i \lesssim 1$, use **sequential BOOT**—merge steps one-by-one.
2. For $\lambda_i \gg 1$, use **vanilla one-shot distillation**—merge all at once.
3. For ambiguous or intermediate $\lambda_i$, run the DP planner to find the optimal schedule [2505.16024].

## 6. Theoretical Significance and Connections

The operator-merging interpretation of multistep distillation formalizes, in the linear regime, why and how different trade-offs between speed (number of student steps) and generative fidelity arise. Each step of a DDIM teacher is a diagonal projection/rescaling, and the only feasible student is a convex combination thereof, subject to signal loss due to finite merging. Minimizing the resultant $W_2$ error via DP not only explains the empirical findings of prior methods—including sequential reduction (BOOT), one-shot distillation, and progressive halving schemes—but reveals the presence of a nontrivial “phase diagram” controlled by the data’s covariance spectrum.

This framework also clarifies why standard progressive distillation [2202.00512], consistency models [2403.06807], and trajectory-matching approaches perform variably across tasks: the regime dictated by data variance directs optimal schedule design for multistep-to-few-step compression. The approach is agnostic to network class or loss, highlighting a fundamental property of the denoising trajectory rather than any architecture-specific effect [2505.16024].

## 7. Outlook and Open Directions

Operator-merging analysis currently applies primarily to linear Gaussian approximations, but suggests extensibility to more general (e.g., non-Gaussian, nonlinear) generative processes by considering the action of nonlinear denoising operators on higher moments or local projections. Automatic merge-schedule planners, based on empirically estimated data covariance, can be deployed to adapt step-reduction strategies for arbitrary datasets or conditional distributions. Extensions to high-dimensional structured output spaces and richer noise schedules represent concrete directions for future work.

The operator-merging view of multistep diffusion model distillation thus provides a unified understanding of the available distillation regimes, enables principled speed/fidelity trade-offs, and offers a sound basis for algorithmic innovation [2505.16024].

Source: https://www.emergentmind.com/topics/multistep-distillation-of-diffusion-models