IDLM: Accelerated Discrete Diffusion LMs

• The paper introduces IDLM, a framework that distills discrete diffusion language models to drastically reduce reverse diffusion steps while preserving generation quality.
• It leverages gradient-stable relaxations and simplex output approximations to address the challenges of backpropagation through discrete one-hot token spaces.
• Empirical results demonstrate 4× to 256× inference acceleration on benchmarks while maintaining teacher-level entropy and generative perplexity.

Inverse-distilled Diffusion LLMs (IDLM) are a distillation framework for discrete Diffusion LLMs (DLMs) introduced to reduce the inference cost of reverse diffusion while preserving generation quality, entropy, and generative perplexity. The method extends Inverse Distillation, originally developed for continuous diffusion models, to the discrete language setting, where tokens are treated as one-hot vectors and reverse-time generation is defined by a learned backward dynamics. The central claim is twofold: first, the inverse objective in the discrete setting can be given a uniqueness guarantee; second, with gradient-stable relaxations, the resulting student generators can reduce the number of inference steps by large factors while retaining the teacher’s distributional behavior (Li et al., 22 Feb 2026).

1. Problem setting and motivation

DLMs have recently achieved strong results in text generation, but their practical deployment is constrained by slow sampling. The underlying bottleneck is that reverse diffusion in text typically requires hundreds to thousands of steps at inference time. IDLM is formulated specifically to distill a pretrained DLM teacher into a faster student generator that preserves the teacher’s generation quality, entropy, and perplexity under substantially smaller step budgets (Li et al., 22 Feb 2026).

The discrete setup represents tokens as one-hot vectors

$$V = \{x \in \{0,1\}^N \colon \sum_{i=1}^N x_i = 1\}, \qquad \Delta = \{ z \in \mathbb{R}^N \colon z \ge 0,\ \langle z,\vec{1}\rangle = 1\}.$$

The forward diffusion process is a continuous-time Markov chain,

$\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$

with conditional forward noising distribution

$$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$

Within this formulation, the target of distillation is not merely a local approximation to a teacher transition rule. The stated goal is to compress the teacher into a student distribution $p_\theta$ whose induced generations preserve key distributional quantities, especially average sequence entropy and GenPPL. This places IDLM at the intersection of discrete diffusion, distribution matching, and fast text generation.

2. Relation to existing discrete diffusion objectives

IDLM is constructed to operate over standard discrete DLM training objectives rather than replacing them with an unrelated proxy. Three formulations are reviewed explicitly: SEDD, MDLM, and UDLM/Duo (Li et al., 22 Feb 2026).

For SEDD, Score Entropy Discrete Diffusion learns a score function $\widehat{s}(x_t,t)$ using

$$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$

The optimum recovers the desired probability ratios.

For MDLM, the model predicts a reverse conditional $\widehat{x}_0(x_t,t)$ and is trained with

$$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$

which is stated to correspond to a negative ELBO objective.

For UDLM/Duo under uniform diffusion, the formulation introduces a Gaussianized latent relaxation

$$\tilde{p}_{t\mid 0}(w_t \mid x_0) = N(\tilde{\alpha}_t x_0, (1-\tilde{\alpha}_t^2)I),$$

with discrete recovery

$$x_t(w_t) \vcentcolon= \arg\max(w_t), \qquad x_t(w_t) \sim p_{t|0}(x_t \mid x_0),$$

and soft relaxation

$\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$0

The associated loss is

$\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$1

These formulations matter because IDLM does not define a separate language-modeling criterion. Instead, it reuses the discrete teacher loss $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$2 and asks for a student distribution under which the teacher remains optimal. This design anchors IDLM to established DLM objectives while redirecting them toward inference acceleration.

3. Inverse distillation in the discrete setting

The core inverse-distillation construction begins from the continuous-diffusion idea of fitting a student distribution $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$3 such that a fixed teacher $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$4 remains optimal under the student’s data distribution. In the discrete language setting, IDLM replaces the continuous teacher loss with a discrete DLM loss and defines

$\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$5

The student is a generator $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$6, later relaxed to $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$7, inducing $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$8 (Li et al., 22 Feb 2026).

A recurring issue in inverse distillation is that nonnegativity alone is insufficient: an objective may admit spurious global minimizers that do not recover the target data distribution. IDLM is designed specifically to avoid that failure mode. In this sense, its defining feature is not only acceleration but also a proof that the inverse formulation is aligned with the true discrete data distribution.

This also clarifies a potential misconception. IDLM is not presented as a heuristic few-step sampler trained only to mimic teacher outputs at selected times. Its objective is a distributional inverse-distillation criterion, and the theoretical analysis is intended to show that minimizing this criterion corresponds to matching $\frac{dp_t}{dt} = Q_t p_t, \quad p_0 = p^*,$9, not merely matching a subset of teacher trajectories.

4. Uniqueness theorem and path-measure interpretation

A major theoretical contribution is the claim that for SEDD, MDLM, and Duo in the $$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$0 limit, the inverse objective is not only nonnegative but uniquely minimized by the true data distribution. The stated result is

$$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$1

and

$$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$2

This excludes spurious zero-loss solutions and makes the optimization target principled in the stated sense (Li et al., 22 Feb 2026).

The proof is organized through reverse-time path measures. Let $$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$3 denote the reverse-time path measure induced by the student distribution $$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$4, and $$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$5 the teacher path measure induced by $$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$6. Then

$$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$7

Using a chain-rule decomposition over path measures,

$$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$8

Since the conditional path term is nonnegative, the lower bound by $$p_{t \mid 0}(x_t \mid x_0) \vcentcolon= \text{Cat}(x_t; \exp(\bar{\sigma}_t Q) x_0), \qquad \bar{\sigma}_t \coloneqq \int_0^t \sigma_s\,ds.$$9 follows immediately. Zero loss implies $p_\theta$0, and hence $p_\theta$1.

For SEDD, the equivalence is made explicit by showing that

$p_\theta$2

The derivation uses the convex variational identity

$p_\theta$3

with maximizer

$p_\theta$4

The parameterization

$p_\theta$5

then connects the KL objective back to the discrete SEDD loss. The corresponding conclusions for MDLM and UDLM are transferred via the stated equivalences under appropriate diffusion matrices.

5. Optimization difficulties and gradient-stable relaxations

Although the inverse objective has a clean theoretical form, optimization in discrete space is described as difficult for two reasons. First, the student generator outputs one-hot tokens in $p_\theta$6, which makes backpropagation awkward and often requires unstable tricks such as hard Gumbel-Softmax. Second, sampling $p_\theta$7 depends on $p_\theta$8, and thus implicitly on the student parameters $p_\theta$9, so differentiation through the sampling procedure is difficult (Li et al., 22 Feb 2026).

The first stabilization replaces the discrete output space by the simplex: $\widehat{s}(x_t,t)$0 This is justified because the discrete losses use $\widehat{s}(x_t,t)$1 only through inner products $\widehat{s}(x_t,t)$2 or through

$\widehat{s}(x_t,t)$3

both of which extend naturally to convex combinations. The student can therefore use differentiable softmax outputs rather than hard discrete samples.

The second stabilization depends on the diffusion formulation. For Duo, the Gaussian latent admits the reparameterization

$\widehat{s}(x_t,t)$4

which isolates randomness in $\widehat{s}(x_t,t)$5, independent of $\widehat{s}(x_t,t)$6. The relaxed objective becomes

$\widehat{s}(x_t,t)$7

For MDLM under the absorbing process, the forward state is either unchanged, $\widehat{s}(x_t,t)$8, or masked, $\widehat{s}(x_t,t)$9. Under the subs parameterization, the non-mask case is trivial because $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$0, giving zero loss, so only the masked state contributes. The masking probability

$$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$1

is independent of $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$2, and the masked state itself is fixed. Consequently, no differentiation through the forward sampling step is required in that case.

Training uses alternating optimization between a fake model $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$3 and the student generator $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$4. Both are initialized from the pretrained teacher $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$5. With the student fixed, the fake model minimizes the teacher loss on student-generated inputs: $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$6 With the fake model fixed, the student update optimizes

$$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$7

The stated pseudocode alternates these two updates for $$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$8 iterations.

6. Multistep students and empirical results

Single-step distillation is reported to remain hard for text, so IDLM is extended to multistep students. The student is initialized from the teacher, uses the same latent or input space as the teacher, and is trained to perform multi-step reverse diffusion. For SEDD and MDLM,

$$L_{\text{SEDD}}(\widehat{s}, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t|x_0)} \biggl[\sum_{y \neq x_t} \lambda_{x_ty} \biggl(\langle \widehat{s}(x_t,t), y\rangle - \frac{p_{t \mid 0}(y \mid x_0)}{p_{t \mid 0}(x_t \mid x_0)} \log \langle \widehat{s}(x_t, t), y\rangle \biggr)\biggr]dt.$$9

and for Duo,

$\widehat{x}_0(x_t,t)$0

This design allows the student to exploit supervision from intermediate teacher states and improves quality at small step budgets (Li et al., 22 Feb 2026).

Experiments are conducted on OpenWebText for the main text generation studies, with One Billion Words used as additional preprocessing or data context for discrete diffusion setups. Teachers are distilled from pretrained SEDD, MDLM, Duo, and Duo-DCD checkpoints; for SEDD, only an absorbing-process checkpoint is available. The student models have about 169M parameters, with 12 layers, 12 heads, hidden size 768, sequence length 1024, and dropout 0.1. Optimization uses AdamW, batch size 8 per device, global batch size 512, constant learning rate with 2500 warmup steps, EMA decay 0.9999, and float64 sampling for evaluation because of GenPPL sensitivity. The reported metrics are GenPPL, defined as GPT-2 Large generative perplexity where lower is better, and average sequence entropy, where higher is better. Baselines include SDTT, Duo-DCD, the teacher models themselves, and the original SEDD, MDLM, and Duo models at various step counts.

The main quantitative findings are summarized below.

Setting	Step reduction	Selected reported values
IDLM-SEDD	$\widehat{x}_0(x_t,t)$1 = $\widehat{x}_0(x_t,t)$2	Matches SEDD quality/diversity; outperforms SEDD in GenPPL across step counts
IDLM-MDLM	$\widehat{x}_0(x_t,t)$3 = $\widehat{x}_0(x_t,t)$4	16 steps: GenPPL 32.74, entropy 5.42; teacher MDLM at 1024 steps: GenPPL 41.29, entropy 5.28
IDLM-MDLM	32 steps	GenPPL 20.37, entropy 5.23
IDLM-Duo (ancestral)	$\widehat{x}_0(x_t,t)$5 = $\widehat{x}_0(x_t,t)$6	GenPPL 63.10, entropy 5.54
IDLM-Duo (greedy)	$\widehat{x}_0(x_t,t)$7 = $\widehat{x}_0(x_t,t)$8	GenPPL 54.05, entropy 5.49
IDLM-DCD (ancestral)	$\widehat{x}_0(x_t,t)$9 = $$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$0 versus Duo-DCD teacher	GenPPL 53.55, entropy 5.41
IDLM-DCD (greedy)	4 steps; $$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$1 over original Duo in the strongest setting	GenPPL 77.49, entropy 5.28

Across these results, the headline statement is that IDLM reduces inference steps by $$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$2 to $$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$3 while preserving the teacher model’s entropy and generative perplexity. In the Duo and Duo-DCD comparisons, the reported accelerations extend to $$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$4 and $$L_{\text{MDLM}}(\widehat{x}_0, x_0) \vcentcolon= \int_0^1 E_{p_{t|0}(x_t \mid x_0)} \bigl[ \lambda_{t} \langle \log\widehat{x}_0(x_t, t), x_0 \rangle \bigr] dt,$$5 in the listed settings. IDLM-SEDD reduces steps from 1024 to 256 while maintaining strong GenPPL and high entropy. IDLM-MDLM reduces 1024 steps to 16, matches MDLM and SDTT quality and diversity in the low-step regime, and is reported to outperform SDTT there. For Duo-family teachers, both ancestral and Greedy-Tail sampling are evaluated; IDLM-DCD is typically strongest in GenPPL, while IDLM-Duo can exhibit higher entropy but slightly worse GenPPL than Duo-DCD in the low-step regime.

7. Limitations, scope, and implications

Several limitations are stated explicitly. GenPPL and entropy are not complete measures of generative capability. Evaluation is mostly on GPT-2-scale models and OpenWebText-style benchmarks. Stronger conclusions would require larger models and more standardized downstream evaluation. One-step generation remains difficult, which is why multistep students are emphasized in practice. The reported gains also depend on available teacher checkpoints and on the specific discrete diffusion formulations being distilled (Li et al., 22 Feb 2026).

Within that scope, the main implications are clear. IDLM shows that inverse distillation can be extended to discrete diffusion LLMs, that the resulting objective is uniquely minimized by the true data distribution for the stated families of objectives, and that simplex relaxation together with reparameterized or structure-aware training makes the method practical. This suggests a route by which diffusion LLMs can be accelerated substantially without abandoning the diffusion framework itself. A plausible implication is that IDLM is most naturally interpreted as a compression method for preserving a teacher’s distributional behavior—rather than as a replacement for diffusion modeling—especially in regimes where quality, diversity, and controllability remain important.