One-by-One Distillation in Diffusion Models

• One-by-One Distillation is a method that compresses multi-step diffusion models into a single-pass generator using teacher-feature drifting.
• The approach leverages a drifting loss in the teacher’s hidden representation space to achieve fixed-point regression without extra score networks or adversarial modules.
• Empirical results show competitive FID scores and faster convergence on benchmarks like ImageNet-64×64 and SDXL, confirming its efficiency in one-step generation.

Searching arXiv for papers on one-step diffusion distillation and related methods. In recent diffusion-model work, “one-by-one distillation” denotes one-step distillation: the compression of a pretrained multi-step diffusion or flow-matching teacher into a single-pass student generator that maps noise directly to an image or latent in one forward pass (Zhang et al., 8 May 2026). The central problem is to preserve the teacher’s image quality, diversity, and conditional fidelity while eliminating the many denoising iterations normally required at inference time. Recent formulations differ chiefly in how they define the student’s training objective. “Teacher-Feature Drifting” (TFD) recasts one-step distillation as fixed-point regression in the frozen teacher’s hidden representation space (Zhang et al., 8 May 2026), whereas VarDiU formulates the problem through a variational upper bound on a diffusive divergence and emphasizes unbiased gradient estimation (Wang et al., 28 Aug 2025).

1. Problem setting and formal objective

A one-step student is typically written as an implicit generator. In TFD, the student is $f_\theta$ with noise input $\epsilon \sim p_\epsilon$ and output $x=f_\theta(\epsilon)$ (Zhang et al., 8 May 2026). In VarDiU, the student is written as

$$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$

where $z\sim \mathcal N(0,I)$ and $g_\theta$ is a neural net (Wang et al., 28 Aug 2025). In both cases, the operational target is the same: distill a teacher that ordinarily requires many denoising evaluations into a generator that produces samples in one pass.

The motivating systems constraint is explicit. Sampling from pretrained diffusion and flow-matching models typically requires many forward passes, while the distilled student requires only one. TFD reports that the student uses one forward pass at teacher scale, approximately a U-Net, whereas the teacher typically uses $50$–$511$ passes (Zhang et al., 8 May 2026). This establishes one-by-one distillation as a latency- and compute-reduction problem, not merely a model-compression problem.

A recurrent methodological issue is the choice of divergence or matching signal. Standard DDPM training minimizes

$$\mathbb E_{x,\epsilon,t}\bigl[\|\epsilon-\epsilon_\theta(x_t,t)\|^2\bigr],$$

which is a pointwise score match (Zhang et al., 8 May 2026). By contrast, distillation methods such as VarDiU begin from the “Diffusive KL”

$$\mathrm{DiKL}(p_\theta\Vert p_d) =\int_0^1 \omega(t)\,\mathrm{KL}\!\bigl(p_\theta^{(t)}(x_t)\Vert p_d^{(t)}(x_t)\bigr)\,\mathrm dt,$$

with $\epsilon \sim p_\epsilon$0 the pretrained teacher diffusion model and $\epsilon \sim p_\epsilon$1 the student marginal at noise level $\epsilon \sim p_\epsilon$2 (Wang et al., 28 Aug 2025).

2. Drifting as a distribution-level matching principle

TFD is built on the Drifting Model objective, which defines attraction and repulsion through mean-shift fields in a representation space (Zhang et al., 8 May 2026). For a point $\epsilon \sim p_\epsilon$3, the fields are

$\epsilon \sim p_\epsilon$4

with

$\epsilon \sim p_\epsilon$5

and the combined drifting field is

$\epsilon \sim p_\epsilon$6

The resulting loss forms a fixed-point target and regresses the student toward it: $\epsilon \sim p_\epsilon$7

The distinctive claim of this formulation is that it performs distribution-level matching through an attraction-repulsion field rather than through an explicit score network or Jacobian-vector products. The paper states this contrast directly: drifting enforces distribution-level matching, and no explicit score network or JVPs are needed (Zhang et al., 8 May 2026). This matters because many one-step distillation pipelines have relied on auxiliary networks, multiple training stages, or adversarial components. TFD’s contribution is to show that a single drifting loss can be used directly for one-step distillation when the representation space is chosen appropriately.

A common misconception is that drifting necessarily requires an independently pretrained feature extractor. TFD’s central observation is that the pretrained diffusion teacher itself already provides a strong representation space, so the original dependence on an additional feature network can be removed (Zhang et al., 8 May 2026).

3. Teacher-feature drifting

The defining move in TFD is to replace pixel-space drifting with drifting in the frozen teacher’s hidden activations (Zhang et al., 8 May 2026). The teacher features are denoted $\epsilon \sim p_\epsilon$8 or $\epsilon \sim p_\epsilon$9, depending on the layer representation. For ImageNet-$x=f_\theta(\epsilon)$0, the chosen layers are U-Net encoder blocks $x=f_\theta(\epsilon)$1 and $x=f_\theta(\epsilon)$2, the bottleneck block, and decoder blocks $x=f_\theta(\epsilon)$3 and $x=f_\theta(\epsilon)$4. For SDXL, the chosen layers are $x=f_\theta(\epsilon)$5, $x=f_\theta(\epsilon)$6, and $x=f_\theta(\epsilon)$7. Each activation is average-pooled over spatial dimensions to yield a fixed-dimensional vector of size $x=f_\theta(\epsilon)$8 (Zhang et al., 8 May 2026).

The layerwise TFD loss is

$x=f_\theta(\epsilon)$9

where $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$0 is the generated image or latent (Zhang et al., 8 May 2026). The empirical claim is that the teacher’s hidden states induce a rich semantic manifold, and drifting in that space preserves semantically meaningful feature geometry without introducing an extra encoder.

TFD also perturbs the inputs to the teacher features: $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$1 with $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$2 (Zhang et al., 8 May 2026). The stated rationale is feature-noise smoothing: mild Gaussian noise smooths high-frequency variations and yields more stable drifting fields. This suggests that the feature geometry used for matching is deliberately regularized before the mean-shift field is estimated.

4. Student architecture, coverage regularization, and training pipeline

The student shares the U-Net structure of the teacher but is executed for a single “step” (Zhang et al., 8 May 2026). Conditioning $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$3, whether a class label or text embedding, is injected via the same attention and time-embedding modules as in the teacher. The forward pass is minimal: sample $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$4, compute $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$5, and treat that output directly as the generated image or, for SDXL, the generated latent (Zhang et al., 8 May 2026).

Because pure drifting can leave “holes” in feature space, TFD augments the drifting objective with a lightweight mode coverage term, the anchor-margin loss (Zhang et al., 8 May 2026). Given anchors $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$6 extracted from real or teacher-sampled positives and generated features $$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$7, the paper defines

$$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$8

$$p_\theta(x_0)=\int \delta(x_0-g_\theta(z))\,p(z)\,\mathrm dz,$$9

Here $z\sim \mathcal N(0,I)$0 controls the aggressiveness of the margin, and $z\sim \mathcal N(0,I)$1 is a kernel bandwidth (Zhang et al., 8 May 2026). The full objective is

$z\sim \mathcal N(0,I)$2

The training pipeline is correspondingly compact. A batch of positives $z\sim \mathcal N(0,I)$3 is sampled from data or teacher; Gaussian inputs produce student outputs $z\sim \mathcal N(0,I)$4; teacher features are extracted from both $z\sim \mathcal N(0,I)$5 and $z\sim \mathcal N(0,I)$6 after noise perturbation; the mini-batch drifting field is estimated at each chosen layer; the anchor-margin term is accumulated from an anchor bank; and $z\sim \mathcal N(0,I)$7 is updated by descending $z\sim \mathcal N(0,I)$8 (Zhang et al., 8 May 2026). The paper’s stated simplification is explicit: no additional score network, Jacobian computations, or adversarial discriminators are required.

5. Empirical characteristics

On ImageNet-$z\sim \mathcal N(0,I)$9 with an EDM teacher and a one-step student, TFD reports a single-step FID of $g_\theta$0, compared with DMD2’s $g_\theta$1 under the same budget (Zhang et al., 8 May 2026). The more prominent empirical difference is convergence speed: TFD reaches $g_\theta$2 in $g_\theta$3 updates, whereas DMD2 needs $g_\theta$4, and it reaches $g_\theta$5 in $g_\theta$6 updates, versus $g_\theta$7 for DMD2 (Zhang et al., 8 May 2026).

On SDXL text-to-image under COCO zero-shot evaluation, TFD reports one-step FID $g_\theta$8 and CLIP score $g_\theta$9, compared with DMD2’s $50$0 and $50$1 (Zhang et al., 8 May 2026). The same evaluation states that visual fidelity is comparable to four-step distilled models and to the $50$2-step teacher. These results support the narrower claim that one-step generation can be competitive in image quality and diversity while substantially simplifying the distillation framework.

The paper also summarizes its contributions as achieving state-of-the-art results for one-step generation on ImageNet-$50$3 and SDXL, with faster convergence (Zhang et al., 8 May 2026). A cautious reading is warranted here: this statement is the paper’s own contribution claim, and its scope is limited to the benchmarks and settings reported there.

6. Relation to variational one-step distillation and open questions

VarDiU provides a contrasting route to one-step diffusion distillation (Wang et al., 28 Aug 2025). Its starting point is that many existing methods approximate the gradient of a diffusive divergence through the student’s score function learned by denoising score matching, and that DSM training is imperfect, so the resulting gradient estimate is biased. VarDiU introduces a variational posterior $50$4 and constructs a variational upper bound

$50$5

with an unbiased gradient estimator obtained by reparameterizing $50$6 (Wang et al., 28 Aug 2025).

The methodological contrast can be summarized as follows.

Method	Core training signal	Stated simplification or advantage
TFD	Drifting field in frozen teacher hidden states	No extra feature extractor; no additional score network, Jacobian computations, or adversarial discriminators (Zhang et al., 8 May 2026)
VarDiU	Variational diffusive upper bound with posterior $50$7	Unbiased gradient estimator; more efficient and stable training than Diff-Instruct (Wang et al., 28 Aug 2025)

VarDiU reports, on a 2D Mixture of 40 Gaussians, that VarDiU-NSF achieves the best values among the listed settings in several comparisons, and that in wall-clock time it reaches Diff-Instruct’s best MMD in approximately $50$8 of the compute (Wang et al., 28 Aug 2025). This establishes a second active line within one-by-one distillation: rather than simplifying the representation side, it addresses bias in the optimization side.

The open questions identified by TFD remain narrow but consequential. The method has so far been demonstrated only with U-Net teachers; behavior under DiT transformers remains unexplored. There is no theoretical recipe yet for selecting feature layers, noise levels $50$9, or kernel radii. The experiments are limited to $511$0 and SDXL scales, leaving larger images, more classes, and other domains for future work (Zhang et al., 8 May 2026). A plausible implication is that one-by-one distillation has moved from proof-of-concept toward recipe design, where representation choice, divergence design, and mode-coverage control are now the central variables.