GDSD: Guided Denoiser Self-Distillation

• GDSD is a diffusion-based self-distillation method that replaces direct teacher-student matching with a guided, denoised target to enhance compatibility.
• In visual applications, the process uses a teacher-guided reverse diffusion to denoise student features, leading to improved performance over traditional KD methods.
• For language models, GDSD adopts a self-teacher from reverse-KL regularized RL, mitigating ELBO biases and ensuring more stable training.

Searching arXiv for the specified GDSD papers to ground the article in the latest records. arXiv Search Query: (Wang et al., 2 Feb 2026) arXiv Search Query: (Tang et al., 28 May 2026) Guided Denoiser Self-Distillation (GDSD) denotes a class of diffusion-based distillation procedures in which the supervisory target is produced by a guided denoising process rather than by direct alignment alone. In current arXiv usage, the term appears in two technically distinct settings: a visual-recognition formulation in which a teacher classifier guides the sampling process of denoising student features through a light-weight diffusion model, and a reinforcement-learning formulation for diffusion LLMs in which the denoiser is distilled from an advantage-guided self-teacher derived from the closed-form optimum of reverse-KL regularized RL (Wang et al., 2 Feb 2026, Tang et al., 28 May 2026). The shared motif is the replacement of naive teacher-student matching with a denoised or self-generated target that is intended to preserve compatibility with the student feature manifold or denoiser parameterization.

1. Conceptual scope and motivating problems

The visual-recognition formulation starts from a standard objection to conventional knowledge distillation: existing KD methods often align feature information between teacher and student by exploring meaningful feature processing and loss functions, but the student model may learn incompatible information from the teacher because of the difference in feature distributions between the teacher and student. To address this, the proposed method performs teacher-guided student Diffusion Self-KD, dubbed DSKD, and explicitly operates instead of the direct teacher-student alignment (Wang et al., 2 Feb 2026).

The diffusion-language-model formulation addresses a different failure mode. It begins from the observation that RL for diffusion LLMs is hindered by the intractability of the policy likelihood, and that a dominant family of methods replaces the likelihood in standard RL with its evidence lower bound (ELBO), estimated from randomly masked sequences. Although these ELBO-based approaches are well aligned with pre-training, they introduce bias through training-inference mismatch by using the ELBO as a likelihood surrogate, which can degrade performance (Tang et al., 28 May 2026).

Taken together, these two lines of work suggest a common interpretation of GDSD: guidance is used to construct a denoised supervisory target that is better matched to the model being trained. In the vision setting, the target is a denoised student feature that could be regarded as a teacher role. In the language-model setting, the target is a closed-form self-teacher denoiser defined by advantage reweighting and reverse-KL regularization.

2. Teacher-guided denoising of student features

In the visual-recognition formulation, GDSD is built on a standard DDPM forward process applied to a clean feature vector or map $x_0 \sim q(x_0)$ over $T$ steps with noise schedule $\{\beta_t\}_{t=1}^T$:

$$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$

with closed-form marginal

$$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$

or equivalently

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$

A neural network $\epsilon_\theta(x_t,t)$ is then trained to recover $\epsilon_t$ by minimizing

$$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$

The distinctive step is the classifier-guided reverse process. A fixed, pre-trained teacher classifier $p_\phi(y \mid x)$ is introduced, and conditioning on the true label $T$0 yields

$T$1

Under a small-variance approximation and first-order Taylor expansion of $T$2, this is approximated by a Gaussian whose mean is shifted by $T$3, where $T$4 is a guidance strength scalar (Wang et al., 2 Feb 2026).

This reverse process is applied not to teacher features but to the student’s intermediate feature $T$5, which is treated as the noisy starting point $T$6. Running the reverse chain from $T$7 to $T$8 produces a denoised feature $T$9. The stated rationale is that the denoised student features encapsulate teacher knowledge and could be regarded as a teacher role, while remaining in the student’s own representational regime. The paper’s summary states that this eliminates discrepancies in mapping manners and feature distributions between the teacher and student while learning meaningful knowledge from the teacher.

3. Backbone design and feature-distillation losses in visual recognition

The visual implementation combines the U-Net design and Diffusion Transformer (DiT) conditioning. The input and output are feature maps of shape $\{\beta_t\}_{t=1}^T$0. Each discrete timestep $\{\beta_t\}_{t=1}^T$1 is embedded via sinusoidal or learned positional encoding and then passed through an MLP to produce scale and shift parameters for batch-normalization layers in each block, together with residual-block scaling factors. The downsampling path uses a $\{\beta_t\}_{t=1}^T$2 convolution with stride $\{\beta_t\}_{t=1}^T$3 to double channels and halve spatial dimensions, followed by residual transformer blocks with time-modulated BN and attention. The upsampling path mirrors this structure with a $\{\beta_t\}_{t=1}^T$4 transposed convolution with stride $\{\beta_t\}_{t=1}^T$5, skip-connections from the encoder, and a final $\{\beta_t\}_{t=1}^T$6 convolution projecting to $\{\beta_t\}_{t=1}^T$7 channels to produce the residual noise estimate $\{\beta_t\}_{t=1}^T$8. The typical depth is given as $\{\beta_t\}_{t=1}^T$9–$$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$0 down/up blocks, each block including $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$1–$$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$2 transformer layers and $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$3 self-attention layer, with exact numbers tuned per dataset (Wang et al., 2 Feb 2026).

Distillation between the original and denoised student features is performed with an LSH-guided feature distillation method. Let $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$4 and $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$5 have the same shape. Global vectors $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$6 and $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$7 are obtained by average-pooling. The local term is a pixel-wise squared $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$8 loss between $$q(x_t \mid x_{t-1}) = \mathcal N\bigl(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),$$9 and $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$0. The global term samples a random Gaussian projection matrix $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$1 and bias $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$2, computes a binary hash code from $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$3 through a sign function, computes prediction probabilities from $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$4 through a sigmoid, and minimizes binary cross-entropy. The combined self-distillation loss is

$$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$5

The overall training objective adds the task loss, diffusion loss, and a conventional KD term:

$$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$6

Here $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$7 is cross-entropy on student logits and $$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$8 is a KL term between student and teacher logits. In practice all four terms are summed and back-propagated (Wang et al., 2 Feb 2026).

4. Self-teacher derivation for diffusion LLMs

In the diffusion-language-model setting, GDSD begins from reverse-KL regularized policy optimization:

$$q(x_t \mid x_0) = \mathcal N\bigl(x_t;\,\sqrt{\bar\alpha_t}\,x_0,\,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t = \prod_{s=1}^t (1-\beta_s),$$9

where $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$0 is the advantage of sequence $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$1, $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$2 is the guidance coefficient, $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$3 is a regularization weight, $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$4 is the current or old policy, and $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$5 is a frozen reference model. The optimal marginal policy over clean sequences has the closed form

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$6

Under the usual masked-diffusion forward process $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$7, this induces a token-level energy-guided denoising distribution

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$8

where $$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon_t, \qquad \epsilon_t \sim \mathcal N(0,I).$$9 is the geometric interpolation of old and reference denoisers, and

$\epsilon_\theta(x_t,t)$0

is the log-partition constant at diffusion time $\epsilon_\theta(x_t,t)$1 (Tang et al., 28 May 2026).

This $\epsilon_\theta(x_t,t)$2 is the self-teacher denoiser. The formulation does not require a separate external teacher policy; instead, the teacher is generated by combining the old denoiser, the reference denoiser, and advantage reweighting. This is the sense in which the method reduces RL to self-distillation.

5. Normalization-free logit matching and training algorithm

Once $\epsilon_\theta(x_t,t)$3 is available in closed form, GDSD trains the student denoiser $\epsilon_\theta(x_t,t)$4 by matching its logits to those of the self-teacher. The raw objective is written in log-probability space as a squared difference between $\epsilon_\theta(x_t,t)$5 and $\epsilon_\theta(x_t,t)$6. Because $\epsilon_\theta(x_t,t)$7 is intractable, the method exploits the translation invariance of the softmax: matching logits up to an additive constant is sufficient (Tang et al., 28 May 2026).

Two normalization-free practical formulations are proposed. The first is direct unnormalized matching, which omits $\epsilon_\theta(x_t,t)$8 entirely. The second is Token-Level Logit Centralization (TLC), which subtracts from each token’s logit its mean over the vocabulary. With centralized teacher logits, the practical GDSD loss becomes

$\epsilon_\theta(x_t,t)$9

This objective is described as normalization-free and likelihood-free from the RL point of view.

The training algorithm proceeds by initializing the student $\epsilon_t$0, setting $\epsilon_t$1, and keeping the reference fixed. For each RL iteration, a batch of prompts is sampled off-policy from $\epsilon_t$2 with iterative re-masking; rewards and advantages are computed; for each sampled pair $\epsilon_t$3, diffusion times are drawn uniformly, masked versions $\epsilon_t$4 are sampled from the forward process, and the denoiser logits of the $\epsilon_t$5, old, and reference models are evaluated. Teacher logits are formed as

$\epsilon_t$6

optionally after TLC, and the student is updated by gradient descent on the GDSD loss. Every $\epsilon_t$7 iterations, the old model is refreshed from the student (Tang et al., 28 May 2026).

6. Relation to ELBO-based RL and stated pathologies

A central claim of the language-model paper is that most existing RL methods for diffusion LLMs insert the ELBO surrogate into policy gradient or PPO, forming importance ratios computed from Monte Carlo–estimated ELBOs. The paper identifies two major sources of training-inference mismatch: first, the ELBO is a loose lower bound on true sequence likelihood, producing importance-ratio bias; second, the actual sampling distribution induced by iterative re-masking differs from the mask-reconstruction ELBO policy. These biases are stated to degrade performance and, in examples cited by the paper, can even cause training collapse in SPG and UniGRPO (Tang et al., 28 May 2026).

The theoretical framing is also comparative. Section 5 of the paper states that forward-KL distillation recovers advantage-weighted ELBO methods such as wd1 and DMPO, which suffer from data inefficiency because negative-advantage samples are down-weighted exponentially, while reverse-KL distillation recovers PG-style objectives such as SPG and ESPO, which carry TIM bias. GDSD’s squared-logit loss with TLC is presented as avoiding both pathologies.

A corresponding misconception in the vision setting is that the method is primarily another direct feature-matching scheme. The description explicitly states the opposite: rather than direct teacher-student alignment, the teacher classifier guides the sampling process of denoising student features through a light-weight diffusion model, after which the original and denoised student features are matched through local and LSH-guided losses (Wang et al., 2 Feb 2026). This suggests that, across both domains, GDSD is best understood as a denoiser-target construction strategy rather than a conventional distillation penalty alone.

7. Empirical results, behavior, and implementation considerations

For visual recognition, experiments on CIFAR-100, ImageNet, and ADE20K are summarized as showing consistent gains of $\epsilon_t$8–$\epsilon_t$9 top-1 over state-of-the-art KD and diffusion-based distillation (DiffKD), and the abstract states that DSKD significantly outperforms existing KD methods across various models and datasets. The accompanying interpretation is that GDSD enriches student features with class semantics and preserves compatibility because the denoised features lie in the student’s own feature manifold (Wang et al., 2 Feb 2026). Code is attached in supplementary material.

For diffusion LLMs, the reported evaluation covers planning, math, and coding benchmarks with Dream-7B and LLaDA-8B. On Dream-7B-Instruct, for logical reasoning in zero-shot settings and generation lengths in $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$0, the reported averages are: d1 at $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$1 on Sudoku and $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$2 on Countdown, wd1 at $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$3 and $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$4, ESPO at $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$5 and $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$6, GDSD at $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$7 and $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$8, and GDSD with TLC at $$\mathcal L_{\mathrm{Diff}} = \mathbb E_{x_0,\epsilon_t,t}\bigl\|\epsilon_t - \epsilon_\theta(x_t,t)\bigr\|^2.$$9 and $p_\phi(y \mid x)$0. The paper states an improvement of up to $p_\phi(y \mid x)$1 absolute on Sudoku, and training curves are described as more stable than those of ELBO methods (Tang et al., 28 May 2026).

On LLaDA-8B-Instruct, the reported mean results over generation lengths are: Sudoku $p_\phi(y \mid x)$2 for ESPO, $p_\phi(y \mid x)$3 for GDSD, and $p_\phi(y \mid x)$4 for GDSD with TLC; Countdown $p_\phi(y \mid x)$5, $p_\phi(y \mid x)$6, and $p_\phi(y \mid x)$7; GSM8K $p_\phi(y \mid x)$8, $p_\phi(y \mid x)$9, and $T$00; MATH500 $T$01, $T$02, and $T$03; HumanEval $T$04, $T$05, and $T$06; and MBPP $T$07, $T$08, and $T$09. The paper summarizes these as consistent gains up to $T$10 on Sudoku, $T$11 on GSM8K, and $T$12 on HumanEval-Plus, with reward dynamics that are smoother and more monotonic than those of d1, SPG, or ESPO (Tang et al., 28 May 2026).

The practical guidance given for RL is specific. Off-policy logit matching is said to avoid ELBO bias entirely; only minor changes are needed to standard diffusion-RL pipelines, namely collecting denoiser logits instead of computing ELBOs and then applying MSE on logits. TLC is reported to further stabilize training by eliminating logit drift, although in a few cases it may slightly over-fit. The guidance coefficient $T$13 controls the teacher’s sharpness, and in practice $T$14 worked well. The $T$15 term can be decoupled as an external regularizer if tuning sensitivity is a concern. The method requires no architectural changes to the diffusion model, and its wall-clock per-step cost is stated to be comparable to ELBO-based methods because the main cost remains batched inference for $T$16 diffusion times. Code is available at the repository cited by the paper (Tang et al., 28 May 2026).