Generative Distribution Distillation (GenDD)

• GenDD is a knowledge distillation framework that reframes teacher-student alignment as conditional generative modeling of teacher feature distributions.
• It leverages a diffusion model with Split Tokenization to stabilize high-dimensional feature learning while mitigating optimization challenges.
• The supervised variant uses Distribution Contraction to embed semantic information, achieving state-of-the-art results on benchmarks like ImageNet.

Searching arXiv for the core GenDD paper and closely related distillation work to ground the article in current literature. Generative Distribution Distillation (GenDD) is a knowledge distillation framework that recasts distillation as conditional generative modeling rather than discriminative point-wise matching. Instead of forcing a student to directly match a teacher’s logits via KL divergence or match intermediate features via MSE, GenDD conditions on the student representation and trains a diffusion model to generate or reconstruct the teacher representation distribution. In its canonical formulation, the target distribution is the teacher feature distribution conditioned on student features, and the framework is intended for both supervised and unsupervised KD on balanced, imbalanced, and unlabeled data (Cui et al., 19 Jul 2025).

1. Definition and conceptual scope

GenDD starts from the conventional decomposition of an image classifier into a feature extractor $\mathcal{F}(\cdot)$ and a classifier $\mathcal{C}(\cdot)$. Classical KD is written as a multi-task objective combining cross-entropy, KL divergence on logits, and feature-level MSE, with manually tuned weights: $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$ This formulation is explicitly characterized as discriminative, point-wise, and sensitive to the choice of KD loss weights (Cui et al., 19 Jul 2025).

GenDD replaces that point-wise view with a conditional generative view. Given an input $x$, the student produces $\mathcal{F}_{s}(x)$, the teacher produces $\mathcal{F}_{t}(x)$, and the generative model learns the conditional mapping from student features to the teacher-feature distribution. The framework is therefore distributional rather than point-wise in output space. A plausible implication is that GenDD turns KD into a problem of modeling $q(\mathcal{F}_t(x)\mid \mathcal{F}_s(x))$, rather than directly penalizing the discrepancy between two deterministic vectors.

This formulation is motivated by several limitations of KL-based KD. The paper emphasizes hyperparameter sensitivity, dependence on labels or cross-entropy supervision, the restrictiveness of point-wise matching, and the fact that teacher logits can become poor supervision when the distillation data are unlabeled or off-domain (Cui et al., 19 Jul 2025). The intended scope is therefore broader than supervised ImageNet-style KD alone.

2. Conditional diffusion formulation

The unsupervised GenDD baseline treats the teacher feature $\hat{x}_0 = \mathcal{F}_t(x)$ as the diffusion target and conditions the denoising model on the student feature $\mathcal{F}_s(x)$. The core training objective is

$$\mathbb{E}_{x, m, \epsilon} \left[ \|\epsilon - \epsilon_{\theta}(\hat{x}_{m}, m, \mathcal{F}_{s}(x)) \|^{2} \right],$$

where $\mathcal{C}(\cdot)$0 is the diffusion step, $\mathcal{C}(\cdot)$1, and $\mathcal{C}(\cdot)$2 is the noisy teacher feature at step $\mathcal{C}(\cdot)$3 (Cui et al., 19 Jul 2025).

The forward corruption follows the standard DDPM form

$\mathcal{C}(\cdot)$4

with

$\mathcal{C}(\cdot)$5

At inference, given $\mathcal{C}(\cdot)$6 and Gaussian initialization $\mathcal{C}(\cdot)$7, the model iteratively denoises from $\mathcal{C}(\cdot)$8 to $\mathcal{C}(\cdot)$9. After sampling $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$0, prediction is obtained by the teacher classifier as

$$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$1

This means the diffusion model reconstructs a teacher-like representation, and the classifier head used for prediction is $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$2, not a separately distilled student classifier (Cui et al., 19 Jul 2025).

A central nuance is that the naive baseline is already a complete conditional diffusion model in feature space, but it encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. The paper gives the concrete example that teacher image representations can be as large as 2048 dimensions, while MAR-style continuous tokenization for image generation uses token dimension around 16. This suggests that direct diffusion on full classifier features is a much harder optimization problem than standard low-dimensional latent diffusion (Cui et al., 19 Jul 2025).

3. Split Tokenization

Split Tokenization is introduced to stabilize the high-dimensional feature-diffusion baseline. Instead of denoising the full teacher feature as a single vector, GenDD divides $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$3 into a sequence of smaller feature tokens: $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$4 Each token is paired with a token position index and the full student feature as conditioning context (Cui et al., 19 Jul 2025).

The diffusion objective becomes token-wise: $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$5 This is not an autoregressive token-generation procedure. The paper describes it as token-wise conditional diffusion, where one lower-dimensional teacher token is noised and denoised at a time, with position information preserved through $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$6 (Cui et al., 19 Jul 2025).

The implementation details given are specific. The teacher feature is divided into non-overlapping tokens, the main token dimension is 64, and the naive full-feature diffusion head is a 3-layer MLP. The empirical finding is equally specific: performance remains competitive when token dimension is at most 256, while it collapses when token dimension increases into the $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$7–$$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$8 range, with reported accuracies that can drop to $$\min_{\theta} CE(\mathcal{C}_{s} \circ \mathcal{F}_{s}(x), y) + \gamma_{0} \cdot KL(\mathcal{C}_{t} \circ \mathcal{F}_{t}(x), \mathcal{C}_{s} \circ \mathcal{F}_{s}(x)) + \sum_{i=1} \gamma_{i} \cdot MSE(f_{t}^{i}, f_{s}^{i}).$$9. This is the operational meaning of the “curse of high-dimensional optimization” in the abstract (Cui et al., 19 Jul 2025).

Conceptually, Split Tokenization turns unsupervised KD into a collection of smaller conditional denoising problems. This suggests that the generative KD objective becomes tractable not by changing the diffusion formalism, but by changing the representation granularity on which the diffusion process operates.

4. Distribution Contraction and the supervised variant

Distribution Contraction addresses the second problem identified in the abstract: the lack of semantic supervision from labels. Rather than adding an explicit classification loss on sampled diffusion outputs, the method modifies the reconstruction target itself: $x$0 where $x$1, $x$2 is the class center for class $x$3, and $x$4 controls contraction strength (Cui et al., 19 Jul 2025).

The supervised GenDD objective is then

$x$5

with $x$6 the noisy version of $x$7 (Cui et al., 19 Jul 2025). The target distribution is no longer the raw teacher-feature distribution; it is a contracted version in which each class manifold is pulled toward its centroid. This injects semantics directly into the reconstruction objective.

The paper’s theoretical claim is that GenDD with Distribution Contraction is approximately equivalent, at the gradient level, to a multi-task objective that combines teacher-feature diffusion reconstruction with cross-entropy on the generated feature. The comparison objective is written as

$x$8

Under the assumptions that the teacher is well trained, the teacher classifier is linear, and the class center is identified with classifier weight $x$9, the gradient induced by Distribution Contraction decomposes into a reconstruction term and a class-center pull that approximates the CE gradient (Cui et al., 19 Jul 2025).

The paper explicitly calls this a “gradient-level surrogate for multi-task learning.” In plain terms, the semantic effect of cross-entropy is folded into the reconstruction target, so the method can realize supervised training without explicit classification loss on multi-step sampling image representations. This is a stronger claim than ordinary target smoothing: it says the generative objective itself becomes a surrogate for supervised KD.

5. Empirical regimes and reported performance

GenDD is evaluated on balanced, imbalanced, and unlabeled data. The paper explicitly lists CIFAR-100 and ImageNet as balanced labeled settings, ImageNet-LT as imbalanced, CC3M as non-target unlabeled data, and ImageNet without labels as target unlabeled data (Cui et al., 19 Jul 2025).

The abstract reports two headline numbers. In the unsupervised setting, GenDD performs competitively and significantly surpasses the KL baseline by 16.29\% on the ImageNet validation set. With label supervision, ResNet-50 achieves 82.28\% top-1 accuracy on ImageNet in 600 epochs training, which the abstract describes as a new state-of-the-art (Cui et al., 19 Jul 2025).

The paper’s experimental interpretation is closely tied to the two technical components. Split Tokenization is the mechanism that makes unsupervised feature-generation stable enough to compete with standard KD baselines, while Distribution Contraction is the mechanism that turns a purely reconstructive diffusion loss into an effective supervised objective. A plausible implication is that the reported ImageNet result depends less on a new classifier head or explicit CE design than on reshaping the teacher feature distribution so that class information is already embedded in the denoising target.

Another empirical point is methodological rather than numerical. The naive supervised alternative would require CE on sampled $\mathcal{F}_{s}(x)$0, which means multi-step diffusion sampling inside training. The framework is motivated partly by avoiding that training-time burden (Cui et al., 19 Jul 2025). This places GenDD in a different optimization regime from standard multi-loss KD.

6. Relation to adjacent literature, variants, and limitations

GenDD in the strict sense of conditional generative KD is introduced explicitly in “Generative Distribution Distillation” (Cui et al., 19 Jul 2025). In a broader sense, the term also illuminates several adjacent lines of work that distill datasets, trajectories, or generative processes into samplers rather than fixed synthetic supports.

A direct precursor is DiM, which distills a dataset into a class-conditional generator and uses a model pool to match logits between real and generated images, storing a reusable sampler instead of a finite synthetic set (Wang et al., 2023). GLaD is a bridge rather than a full distribution-distillation method: it re-parameterizes distilled images through a fixed pretrained generator and optimizes latent representations, but it still ultimately produces a finite distilled set rather than an explicit distilled distribution model (Cazenavette et al., 2023). Later dataset-distillation works intensified the generative view in several directions: balancing global structure and local details with a conditional GAN (Li et al., 2024), using SDXL-Turbo as a generator-plus-prompt deployment recipe under a strict time budget (Su et al., 2024), adding min-max representativeness and diversity terms in latent diffusion (Fan et al., 24 Mar 2025), introducing self-adaptive memory for diversity-driven diffusion distillation (Li et al., 26 May 2025), and aligning an off-the-shelf text-to-image model to an “intrinsic core distribution” without target-specific generator training (Zhou et al., 3 Dec 2025).

A second adjacent line extends the same distributional intuition beyond dataset distillation. DDIL recasts few-step diffusion distillation as imitation learning over an aggregate state distribution that mixes forward-diffused data and student-induced reverse trajectories (Garrepalli et al., 2024). “Distillation of Discrete Diffusion by Exact Conditional Distribution Matching” is an unusually explicit example of matching conditional generative distributions in a CTMC setting (Gao et al., 15 Dec 2025). SD3.5-Flash reformulates KL-style distribution matching for rectified flows by evaluating teacher and proxy models on states that lie on the student trajectory rather than on arbitrary re-noised endpoints (Bandyopadhyay et al., 25 Sep 2025). DD3G transfers a multi-view diffusion model into a fast 3D Gaussian generator through deterministic DDIM supervision and an SDS-style verifier, making it a cross-domain instance of generative distribution distillation (Qin et al., 1 Apr 2025). RM-Distiller applies a closely related idea to reward modeling by distilling the teacher LLM’s token-level next-token distribution through NLL and KL regularization (Zhou et al., 20 Jan 2026).

This broader literature also clarifies what GenDD is not. Some papers are “generative distillation” only in a weak sense. For example, the ECCV 2024 challenge method based on SDXL-Turbo uses a pretrained diffusion model with class-name prompts and post data augmentation, but it “does not require any parameter optimization” and does not learn a new target distribution from scratch; it is therefore only weakly “distribution distillation” in the stronger sense (Su et al., 2024). CoDA makes a related distinction when it argues that many prior generative dataset distillation methods either require a diffusion model pre-trained or fine-tuned on the full target dataset or suffer from a significant distribution mismatch when using off-the-shelf text-to-image models (Zhou et al., 3 Dec 2025).

Within the original GenDD framework itself, the principal limitations are already encoded in its two technical fixes. The naive baseline is unstable in high-dimensional feature spaces, and pure unsupervised reconstruction lacks semantic supervision from labels (Cui et al., 19 Jul 2025). More broadly, the framework depends on teacher features and, at inference, on the teacher classifier $\mathcal{F}_{s}(x)$1. This suggests that GenDD is not merely a replacement for KL loss; it is a reorganization of the teacher-student interface around conditional feature generation.

In that sense, GenDD marks a shift in how KD is conceptualized. The distilled object is no longer just a student logit vector trained to agree with a teacher logit vector. It is a conditional generative process over teacher representations, stabilized by Split Tokenization and semantically shaped by Distribution Contraction, with the explicit claim that such a process can serve as a gradient-level surrogate for conventional multi-task KD (Cui et al., 19 Jul 2025).