Diffusion Self-Distillation

• Diffusion self-distillation is a family of methods where diffusion models act as teachers to supervise faster, adapted student models through various structured paradigms.
• The approach leverages diverse supervision signals such as EMA-based teacher models, frozen offline teachers, and self-generated paired data to align student outputs with the diffusion process.
• Practical applications include one-step and few-step generative acceleration, improved latent space learning, and expandable techniques for non-generative tasks like deep metric learning.

Searching arXiv for papers on diffusion self-distillation and closely related distillation formulations. Diffusion self-distillation denotes a heterogeneous family of procedures in which a diffusion model, a diffusion-defined dynamical system, or a diffusion-structured relation is used to supervise a faster, adapted, or differently parameterized student. Across the literature, the term does not refer to a single canonical algorithm. It includes one-step and few-step generator distillation, latent-space and decoder-level self-distillation, synthetic paired-data bootstrapping, trajectory and denoiser distillation in diffusion language and video models, and, in some papers, graph or manifold diffusion rather than generative denoising (Geng et al., 2023).

1. Scope and terminology

A concise way to organize the literature is by the object being distilled rather than by modality alone.

Family	Distilled object	Representative papers
One-step or few-step acceleration	Noise-to-image map, ODE/SDE transport, diffused marginals	(Geng et al., 2023, Wang et al., 28 Aug 2025, Tee et al., 2024, Jiang et al., 23 Aug 2025)
Representation or data bootstrapping	Clean latents, RGB decoder outputs, synthetic paired datasets	(Wang et al., 18 Nov 2025, Cai et al., 2024, Bahmani et al., 23 Sep 2025)
Trajectory or denoiser distillation	Teacher rollouts, denoiser logits, self-generated rollouts	(Zhang et al., 12 Feb 2026, Tang et al., 28 May 2026, Zheng et al., 24 Jun 2026)
Non-generative diffusion settings	Batch-manifold diffusion, graph diffusion, feature-space diffusion	(Zeng et al., 2022, Zhang et al., 2020, Wang et al., 2 Feb 2026)

A recurring source of confusion is the word “diffusion.” In some papers it denotes denoising diffusion or probability-flow dynamics, whereas in others it denotes graph diffusion or a random-walk-with-restart style process on a batch manifold. "Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning" makes this distinction explicit: its diffusion is a graph/manifold diffusion process applied to pairwise relations inside each mini-batch, not a generative diffusion model (Zeng et al., 2022).

A second ambiguity concerns “self-distillation.” Some methods are best described as online or trajectory-level self-distillation; others are more precisely offline teacher-student distillation from a frozen diffusion teacher. "One-Step Diffusion Distillation via Deep Equilibrium Models" explicitly states that its method is better described as offline teacher-student distillation than as self-distillation, because the teacher is pretrained and frozen and training data are precomputed as paired noise/image examples (Geng et al., 2023).

2. Core supervision patterns

Despite this heterogeneity, several supervision patterns recur. One is explicit teacher-student asymmetry. "Iterative Graph Self-Distillation" uses a teacher-student distillation process in which the teacher model is constructed using an exponential moving average of the student model, and graph pairs are viewed under graph diffusion augmentations (Zhang et al., 2020). In deep metric learning, Online Batch Diffusion-based Self-Distillation uses the model from the previous epoch as the teacher for the current epoch, and the student matches a batch-manifold-refined similarity target rather than raw teacher similarities (Zeng et al., 2022).

A second pattern is frozen-teacher offline supervision. In "One-Step Diffusion Distillation via Deep Equilibrium Models," the teacher is used only to generate a dataset of deterministic noise/image pairs, and the student is trained with a simple pixel-space reconstruction loss; there is no adversarial loss, no perceptual loss, no consistency loss, no score matching loss, and no trajectory matching term (Geng et al., 2023). "Teacher-Feature Drifting" pushes this farther by reusing the pretrained diffusion teacher itself as the representation network: the student is trained in the teacher’s own intermediate hidden-state space rather than through an extra pretrained feature extractor (Zhang et al., 8 May 2026).

A third pattern is self-generated supervisory data. "Diffusion Self-Distillation for Zero-Shot Customized Image Generation" uses a pretrained text-to-image model to generate its own paired dataset: the model creates multi-panel identity-consistent grids, a Visual-LLM curates same-identity pairs, and the original model is then fine-tuned into a text+image-to-image model (Cai et al., 2024). "Lyra" similarly augments a pretrained camera-controlled video diffusion model with a 3DGS decoder that is supervised by the output of the RGB decoder, so that the 3D branch is trained from synthetic data generated by the video diffusion model itself (Bahmani et al., 23 Sep 2025).

These schemes differ in where the teacher signal enters: through EMA targets, frozen endpoint samples, intermediate hidden states, denoiser logits, or rendered views. This suggests that diffusion self-distillation is best understood as a family of supervision interfaces built around a diffusion teacher rather than as a single loss form.

3. One-step and few-step generative acceleration

The most visible strand of the literature compresses many-step sampling into one-step or few-step generation. "One-Step Diffusion Distillation via Deep Equilibrium Models" trains a student to approximate the teacher’s full deterministic sampling map from initial Gaussian noise directly to the final image; the teacher dataset consists of 1M unconditional or conditional pairs generated with Heun’s second-order deterministic solver, and the student is a Generative Equilibrium Transformer trained offline (Geng et al., 2023). On unconditional CIFAR-10, GET-Base reaches FID $7.42$, and an improved-setting variant reaches FID $6.91$; on class-conditional CIFAR-10, GET-Base reaches FID $6.25$, improving to FID $5.66$ with 2M training pairs (Geng et al., 2023).

A different route is variational distribution matching. "VarDiU" defines a Variational Diffusive Upper Bound,

$$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$

which upper-bounds the diffusive reverse KL and yields an unbiased gradient estimator without learning the student score. The paper’s central claim is that DSM-based gradients are biased because the student score is only approximated, whereas VarDiU avoids this by introducing a variational posterior $q_\phi^{(t)}(z\mid x_t)$ (Wang et al., 28 Aug 2025).

Other methods distill the dynamics more directly. "Physics Informed Distillation" treats the teacher as defining a probability flow ODE and trains a student trajectory function by minimizing a PINN-style residual rather than by matching synthetic teacher trajectories. For EDM, the distilled ODE is

$$\frac{\mathrm dx}{\mathrm dt}=\frac{x-D_\phi(x,t)}{t},$$

and the residual is stabilized by rearranging it to

$x-t\frac{dx}{dt}=D_\phi(x,t).$

PID reports FID $3.92$ on CIFAR-10 and FID $9.49$ on ImageNet $6.91$0, and it eliminates the need for synthetic dataset generation during distillation (Tee et al., 2024).

"Sig-DEG for Distillation" instead treats the reverse process as an SDE trajectory-approximation problem on a coarse grid. Its student receives partial signatures of Brownian motion over each coarse interval and learns a recurrent coarse reverse transition. The paper states that the partial signature terms can be simulated exactly at inference time, and reports speedups of $6.91$1 and $6.91$2 on MNIST for 10-step and 5-step students distilled from a 1500-step teacher, with FID $6.91$3 and $6.91$4, respectively (Jiang et al., 23 Aug 2025).

Still another variant is transport self-distillation. "One-Step Diffusion Samplers via Self-Distillation and Deterministic Flow" trains a step-conditioned ODE so that one large probability-flow step reproduces the effect of two smaller ones. Its state-space consistency loss is

$6.91$5

and it couples this with a volume-consistency regularizer and a deterministic-flow importance weight

$6.91$6

The paper argues that standard ELBO estimates degrade badly in the few-step regime because common discrete integrators yield mismatched forward/backward kernels (Jutras-Dube et al., 4 Dec 2025).

"Teacher-Feature Drifting" represents a simpler one-step alternative. It trains the student in the teacher’s hidden-state space with a drifting objective,

$6.91$7

plus an anchor-margin coverage term. It reports FID $6.91$8 on ImageNet-$6.91$9 and $6.25$0 on SDXL while removing the need for an extra representation network (Zhang et al., 8 May 2026).

4. Latent, decoder, and data self-distillation

A separate line of work interprets diffusion training itself as self-distillation. "Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model" argues that latent diffusion has the same structural ingredients as non-contrastive self-distillation and that naive end-to-end joint training fails catastrophically because of latent collapse. The paper defines a unified form

$6.25$1

and replaces velocity prediction with a clean-latent target. Its final self-distillation loss is

$6.25$2

with $6.25$3, $6.25$4, and $6.25$5. The target encoder is updated using EMA, and the paper reports FID $6.25$6 for 42M/118M/205M parameter models on ImageNet $6.25$7 without classifier-free guidance (Wang et al., 18 Nov 2025).

Another interpretation of self-distillation is data bootstrapping. "Diffusion Self-Distillation for Zero-Shot Customized Image Generation" uses a pretrained text-to-image model to generate image grids that often contain the same subject in different contexts, curates paired examples with a VLM, and trains a text+image-conditioned student on about 400k subject-consistent image pairs (Cai et al., 2024). On DreamBench++, the reported overall GPT-based scores for the method are CP $6.25$8, PF $6.25$9, and $5.66$0, with debiased $5.66$1 (Cai et al., 2024).

"Lyra" transposes the same idea into 3D reconstruction. A pretrained camera-controlled video diffusion model provides latent videos and RGB supervision, while a newly introduced 3DGS decoder predicts explicit Gaussians from the same latent representation. The training loss is

$5.66$2

with $5.66$3, $5.66$4, $5.66$5, and $5.66$6 (Bahmani et al., 23 Sep 2025). The paper reports state-of-the-art performance on RealEstate10K, DL3DV, and Tanks and Temples, and stresses that the framework eliminates the need for multi-view training data (Bahmani et al., 23 Sep 2025).

Taken together, these papers show that “self-distillation” may refer not only to faster sampling but also to stabilizing latent learning, turning an RGB decoder into a 3D teacher, or converting a model’s own generations into supervised training pairs.

5. Trajectory and denoiser self-distillation in language and video models

Diffusion language modeling has produced trajectory-level self-distillation methods in which the student is trained on teacher-visited intermediate states rather than on random corruptions. "T3D" defines teacher and student diffusion LLMs $5.66$7 and $5.66$8, collects teacher denoising trajectories, and trains the student on pairs $5.66$9 sampled from the teacher’s trajectory distribution. Its direct trajectory loss is

$$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$0

and the paper’s main variant replaces this with a DDO-based discriminative objective. On SDAR-4B-Chat with TokPS $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$1, the reported average rises from $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$2 for the original model to $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$3 for T3D (Zhang et al., 12 Feb 2026).

A reinforcement-learning variant is "GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion LLMs." Here the distilled object is the denoiser itself. Starting from a reverse-KL-regularized RL objective, the paper derives an advantage-guided teacher denoiser

$$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$4

then reduces RL to a normalization-free denoiser-matching problem through token-level logit centralization (Tang et al., 28 May 2026). The paper reports that on Dream-7B planning tasks, GDSD improves over ELBO-based baselines by up to $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$5 absolute test accuracy (Tang et al., 28 May 2026).

In autoregressive video diffusion, "Causal-rCM" places self-forcing on the on-policy, reverse-divergence side of a broader forward/reverse framework. Teacher-forcing consistency training provides offline initialization, and self-forcing DMD refines the model on its own rollout distribution. The paper’s distilled 2-step causal Wan2.1-1.3B model achieves a VBench-T2V score of $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$6 with only 1 or 2 sampling steps (Zheng et al., 24 Jun 2026). The paper is careful that self-forcing is not an independent self-distillation loss by itself; rather, it is the training regime under which reverse-type distillation is applied (Zheng et al., 24 Jun 2026).

A plausible implication is that diffusion self-distillation becomes increasingly trajectory-centric as one moves from image sampling to language and video generation. In these settings, the central mismatch is often not only sample quality but also the distribution of intermediate states seen at inference time.

6. Related non-generative usages, misconceptions, and recurrent limitations

Some papers use diffusion self-distillation outside generative denoising. "Iterative Graph Self-Distillation" learns graph-level representation in an unsupervised manner through instance discrimination, uses graph diffusion augmentations, and constructs the teacher model using an exponential moving average of the student model; it also extends to semi-supervised training and self-training fine-tuning (Zhang et al., 2020). "Self-distillation with Online Diffusion on Batch Manifolds Improves Deep Metric Learning" uses a batch graph with

$$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$7

to refine teacher similarity targets before distillation, and reports that the full OBD-SD formulation improves MS loss from $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$8 to $$\mathrm{DiU}(\theta,\phi)=\int_0^1 \omega(t)\,\mathrm{U}^{(t)}(\theta,\phi)\,dt,$$9 R@1 on CUB and from $q_\phi^{(t)}(z\mid x_t)$0 to $q_\phi^{(t)}(z\mid x_t)$1 on CARS (Zeng et al., 2022). "Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model" uses a lightweight diffusion model in feature space, teacher-classifier guidance during reverse sampling, and then distills the original student feature toward a denoised student feature that “could be regarded as a teacher role” (Wang et al., 2 Feb 2026).

This breadth explains a common misconception: diffusion self-distillation is not synonymous with one-step diffusion sampling. It can mean few-step sampler compression, latent-space collapse avoidance, synthetic paired-data creation, trajectory distillation, denoiser-level RL, or graph/manifold diffusion on non-generative relations (Geng et al., 2023).

Several limitations recur across the literature. Some are methodological. Offline endpoint distillation in GET requires a deterministic sampler and was evaluated only on CIFAR-10 (Geng et al., 2023). VarDiU’s experiments are restricted to a 2D mixture-of-Gaussians benchmark, and extension to high-dimensional image or video generation is left as future work (Wang et al., 28 Aug 2025). PID is competitive but not state of the art on ImageNet $q_\phi^{(t)}(z\mid x_t)$2, and its training is slower than Progressive Distillation and Consistency Models (Tee et al., 2024). T3D explicitly states that full-step decoding remains superior overall, even though few-step decoding improves substantially (Zhang et al., 12 Feb 2026). OBD-SD does not consistently improve proxy-based deep metric learning (Zeng et al., 2022).

Other limitations concern supervision quality. Data-bootstrapping methods depend on the quality of the teacher’s generations and on auxiliary curation systems such as VLMs or depth estimators (Cai et al., 2024). Lyra’s self-distillation depends on the ability of a camera-controlled video diffusion model to provide multi-view-consistent supervision, although the paper argues that synthetic self-distillation can outperform real multiview data alone (Bahmani et al., 23 Sep 2025). Causal-rCM also stresses that self-forcing still relies on teacher or fake-score guidance and therefore should not be read as teacher-free self-distillation (Zheng et al., 24 Jun 2026).

Across these variants, the unifying idea is not a single loss but a structural asymmetry: a diffusion-defined teacher, teacher proxy, or teacher geometry supplies targets that are easier to match than the original many-step or weakly supervised objective. The student may then be a one-step generator, a latent-space learner, a denoiser, a 3D decoder, or a model trained on its own rollouts. This suggests that diffusion self-distillation is best regarded as a design pattern for converting implicit diffusion knowledge into a more directly usable form.