---
title: One-by-One Distillation in Diffusion Models
url: https://www.emergentmind.com/topics/one-by-one-distillation
type: topic
---

# One-by-One Distillation in Diffusion Models

Searching arXiv for recent papers on one-step diffusion distillation and related methods.
In recent diffusion-model work, “one-by-one distillation” denotes one-step distillation: the compression of a pretrained multi-step diffusion or flow-matching teacher into a single-pass student generator that maps noise directly to an image or latent in one forward pass [2605.07327]. The central problem is to preserve the teacher’s image quality, diversity, and conditional fidelity while eliminating the many denoising iterations normally required at inference time. Recent formulations differ chiefly in how they define the student’s training objective. “Teacher-Feature Drifting” (TFD) recasts one-step distillation as fixed-point regression in the frozen teacher’s hidden representation space [2605.07327], whereas VarDiU formulates the problem through a variational upper bound on a diffusive divergence and emphasizes unbiased gradient estimation [2508.20646].

## 1. Problem setting and formal objective

A one-step student is typically written as an implicit generator. In TFD, the student is $f_\theta$ with noise input $\epsilon \sim p_\epsilon$ and output $x=f_\theta(\epsilon)$ [2605.07327]. In VarDiU, the student is written as
\[
p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,
\]
where $z\sim \mathcal N(0,I)$ and $g_\theta$ is a neural net [2508.20646]. In both cases, the operational target is the same: distill a teacher that ordinarily requires many denoising evaluations into a generator that produces samples in one pass.

The motivating systems constraint is explicit. Sampling from pretrained diffusion and flow-matching models typically requires many forward passes, while the distilled student requires only one. TFD reports that the student uses one forward pass at teacher scale, approximately a U-Net, whereas the teacher typically uses $50$–$511$ passes [2605.07327]. This establishes one-by-one distillation as a latency- and compute-reduction problem, not merely a model-compression problem.

A recurrent methodological issue is the choice of divergence or matching signal. Standard DDPM training minimizes
\[
\mathbb E_{x,\epsilon,t}\bigl[\|\epsilon-\epsilon_\theta(x_t,t)\|^2\bigr],
\]
which is a pointwise score match [2605.07327]. By contrast, distillation methods such as VarDiU begin from the “Diffusive KL”
\[
\mathrm{DiKL}(p_\theta\Vert p_d)
=\int_0^1 \omega(t)\,\mathrm{KL}\!\bigl(p_\theta^{(t)}(x_t)\Vert p_d^{(t)}(x_t)\bigr)\,\mathrm dt,
\]
with $p_d$ the pretrained teacher diffusion model and $p_\theta^{(t)}$ the student marginal at noise level $t$ [2508.20646].

## 2. Drifting as a distribution-level matching principle

TFD is built on the Drifting Model objective, which defines attraction and repulsion through mean-shift fields in a representation space [2605.07327]. For a point $z$, the fields are
\[
V_p^+(z)=\frac{\mathbb E_{y\sim p}[k(z,y)(y-z)]}{\mathbb E_{y\sim p}[k(z,y)]},
\qquad
V_q^-(z)=\frac{\mathbb E_{y\sim q}[k(z,y)(y-z)]}{\mathbb E_{y\sim q}[k(z,y)]},
\]
with
\[
k(z,y)=\exp(-\|z-y\|/\tau),
\]
and the combined drifting field is
\[
V_{p,q}(z)=V_p^+(z)-V_q^-(z).
\]
The resulting loss forms a fixed-point target and regresses the student toward it:
\[
L_{\rm drift}
=
\mathbb E_{\epsilon\sim p_\epsilon}
\Bigl[
\bigl\|
f_\theta(\epsilon)
-
\operatorname{sg}\bigl(f_\theta(\epsilon)+V_{p_{\rm data},q_\theta}(f_\theta(\epsilon))\bigr)
\bigr\|_2^2
\Bigr].
\]

The distinctive claim of this formulation is that it performs distribution-level matching through an attraction-repulsion field rather than through an explicit score network or Jacobian-vector products. The paper states this contrast directly: drifting enforces distribution-level matching, and no explicit score network or JVPs are needed [2605.07327]. This matters because many one-step distillation pipelines have relied on auxiliary networks, multiple training stages, or adversarial components. TFD’s contribution is to show that a single drifting loss can be used directly for one-step distillation when the representation space is chosen appropriately.

A common misconception is that drifting necessarily requires an independently pretrained feature extractor. TFD’s central observation is that the pretrained diffusion teacher itself already provides a strong representation space, so the original dependence on an additional feature network can be removed [2605.07327].

## 3. Teacher-feature drifting

The defining move in TFD is to replace pixel-space drifting with drifting in the frozen teacher’s hidden activations [2605.07327]. The teacher features are denoted $\phi_T^\ell(x,c)$ or $h_T^\ell(\cdot,t_f,c)$, depending on the layer representation. For ImageNet-$64\times 64$, the chosen layers are U-Net encoder blocks $6$ and $11$, the bottleneck block, and decoder blocks $7$ and $12$. For SDXL, the chosen layers are $\texttt{down\_block.2}$, $\texttt{mid\_block}$, and $\texttt{up\_block.0}$. Each activation is average-pooled over spatial dimensions to yield a fixed-dimensional vector of size $D_\ell$ [2605.07327].

The layerwise TFD loss is
\[
L_{\rm TFD}
=
\sum_{\ell\in S}
\mathbb E_{\epsilon,c}
\Bigl[
\bigl\|
\phi_T^\ell(x^{-},c)
-
\operatorname{sg}\bigl(
\phi_T^\ell(x^{-},c)+V_{p_T^\ell,q_T^\ell}(\phi_T^\ell(x^{-},c))
\bigr)
\bigr\|_2^2
\Bigr],
\]
where $x^{-}=f_\theta(\epsilon,c)$ is the generated image or latent [2605.07327]. The empirical claim is that the teacher’s hidden states induce a rich semantic manifold, and drifting in that space preserves semantically meaningful feature geometry without introducing an extra encoder.

TFD also perturbs the inputs to the teacher features:
\[
x_{t_f}=x+\sigma_{t_f}\xi,
\]
with $\sigma_{t_f}\approx 0.1$ [2605.07327]. The stated rationale is feature-noise smoothing: mild Gaussian noise smooths high-frequency variations and yields more stable drifting fields. This suggests that the feature geometry used for matching is deliberately regularized before the mean-shift field is estimated.

## 4. Student architecture, coverage regularization, and training pipeline

The student shares the U-Net structure of the teacher but is executed for a single “step” [2605.07327]. Conditioning $c$, whether a class label or text embedding, is injected via the same attention and time-embedding modules as in the teacher. The forward pass is minimal: sample $\epsilon\sim\mathcal N(0,I)$, compute $x^{-}=f_\theta(\epsilon,c)$, and treat that output directly as the generated image or, for SDXL, the generated latent [2605.07327].

Because pure drifting can leave “holes” in feature space, TFD augments the drifting objective with a lightweight mode coverage term, the anchor-margin loss [2605.07327]. Given anchors $a_i$ extracted from real or teacher-sampled positives and generated features $z_j$, the paper defines
\[
s_i=\frac{1}{M}\sum_{j=1}^{M}\exp(-\|a_i-z_j\|/h),
\qquad
\bar s_i=\frac{1}{M-1}\sum_{k\ne i}\exp(-\|a_i-a_k\|/(2h)),
\]
\[
\rho_i=\alpha\cdot \bar s_i,
\qquad
L_{\rm anchor}=\frac{1}{M}\sum_{i=1}^{M}\max(0,\rho_i-s_i).
\]
Here $\alpha\in[0,1]$ controls the aggressiveness of the margin, and $h$ is a kernel bandwidth [2605.07327]. The full objective is
\[
L=L_{\rm TFD}+\lambda_{\rm anchor}\,L_{\rm anchor}.
\]

The training pipeline is correspondingly compact. A batch of positives $\{x_i^+,c_i\}$ is sampled from data or teacher; Gaussian inputs produce student outputs $x_i^- = f_\theta(\epsilon_i,c_i)$; teacher features are extracted from both $x_i^+$ and $x_i^-$ after noise perturbation; the mini-batch drifting field is estimated at each chosen layer; the anchor-margin term is accumulated from an anchor bank; and $\theta$ is updated by descending $\nabla_\theta(L_{\rm TFD}+\lambda_{\rm anchor}L_{\rm anchor})$ [2605.07327]. The paper’s stated simplification is explicit: no additional score network, Jacobian computations, or adversarial discriminators are required.

## 5. Empirical characteristics

On ImageNet-$64\times 64$ with an EDM teacher and a one-step student, TFD reports a single-step FID of $1.58$, compared with DMD2’s $1.51$ under the same budget [2605.07327]. The more prominent empirical difference is convergence speed: TFD reaches $\mathrm{FID}\le 10$ in $1\text{K}$ updates, whereas DMD2 needs $5.5\text{K}$, and it reaches $\mathrm{FID}\le 3$ in $12.5\text{K}$ updates, versus $24\text{K}$ for DMD2 [2605.07327].

On SDXL text-to-image under COCO zero-shot evaluation, TFD reports one-step FID $=18.39$ and CLIP score $=0.332$, compared with DMD2’s $19.01$ and $0.336$ [2605.07327]. The same evaluation states that visual fidelity is comparable to four-step distilled models and to the $50$-step teacher. These results support the narrower claim that one-step generation can be competitive in image quality and diversity while substantially simplifying the distillation framework.

The paper also summarizes its contributions as achieving state-of-the-art results for one-step generation on ImageNet-$64\times 64$ and SDXL, with faster convergence [2605.07327]. A cautious reading is warranted here: this statement is the paper’s own contribution claim, and its scope is limited to the benchmarks and settings reported there.

## 6. Relation to variational one-step distillation and open questions

VarDiU provides a contrasting route to one-step diffusion distillation [2508.20646]. Its starting point is that many existing methods approximate the gradient of a diffusive divergence through the student’s score function learned by denoising score matching, and that DSM training is imperfect, so the resulting gradient estimate is biased. VarDiU introduces a variational posterior $q_\phi^{(t)}(z\mid x_t)$ and constructs a variational upper bound
\[
\mathrm{DiU}(\theta,\phi)
=
\int_0^1 \omega(t)\,U^{(t)}(\theta,\phi)\,\mathrm dt
\;\ge\;
\mathrm{DiKL}(p_\theta\Vert p_d),
\]
with an unbiased gradient estimator obtained by reparameterizing $x_t=g_\theta(z)+\sigma_t\varepsilon$ [2508.20646].

The methodological contrast can be summarized as follows.

| Method | Core training signal | Stated simplification or advantage |
|---|---|---|
| TFD | Drifting field in frozen teacher hidden states | No extra feature extractor; no additional score network, Jacobian computations, or adversarial discriminators [2605.07327] |
| VarDiU | Variational diffusive upper bound with posterior $q_\phi^{(t)}(z\mid x_t)$ | Unbiased gradient estimator; more efficient and stable training than Diff-Instruct [2508.20646] |

VarDiU reports, on a 2D Mixture of 40 Gaussians, that VarDiU-NSF achieves the best values among the listed settings in several comparisons, and that in wall-clock time it reaches Diff-Instruct’s best MMD in approximately $60\%$ of the compute [2508.20646]. This establishes a second active line within one-by-one distillation: rather than simplifying the representation side, it addresses bias in the optimization side.

The open questions identified by TFD remain narrow but consequential. The method has so far been demonstrated only with U-Net teachers; behavior under DiT transformers remains unexplored. There is no theoretical recipe yet for selecting feature layers, noise levels $\sigma_{t_f}$, or kernel radii. The experiments are limited to $64\times 64$ and SDXL scales, leaving larger images, more classes, and other domains for future work [2605.07327]. A plausible implication is that one-by-one distillation has moved from proof-of-concept toward recipe design, where representation choice, divergence design, and mode-coverage control are now the central variables.

Source: https://www.emergentmind.com/topics/one-by-one-distillation