DiLaDiff: Distilled Latent Diffusion LM

• The paper introduces DiLaDiff, a distilled latent-guided diffusion model that repairs the independence failure of masked token decoding, achieving up to 30% GenPPL improvement and reducing latent overhead to about 5%.
• DiLaDiff is a latent-augmented language model that integrates a continuous semantic latent space to capture long-range dependencies, enabling efficient few-step latent planning.
• The method employs a three-stage architecture with an auto-encoder, latent diffusion prior, and MeanFlow distillation to achieve faster inference without sacrificing generation quality.

DiLaDiff is a distilled latent-augmented diffusion LLM that addresses a central limitation of masked diffusion LLMs: the factorized reverse process used in discrete token-space decoding does not capture correlations between decoded tokens, producing a harsh trade-off between sampling quality and throughput. The method introduces a continuous latent space with semantic capabilities, a latent diffusion prior over that space, and a consistency-style distillation stage that turns the latent prior into a few-step generator. In the paper’s formulation, DiLaDiff is the distilled version of LaDiff, and its purpose is to make latent-guided diffusion language modeling computationally practical while preserving the gains in coherence and controllability afforded by the latent variable (Lemercier et al., 22 May 2026).

1. Problem formulation and the failure mode of masked diffusion decoding

The starting point is the standard masked diffusion LLM (MDLM), which generates text by repeatedly unmasking tokens. This permits parallel generation in principle, but the reverse posterior is factorized across token positions: $$q^\theta_{s\mid t}(\mathbf{x}_s\mid \mathbf{x}_t) := \prod_\ell q_{s\mid t}\!\left(\cdot \mid \mathbf{x}_t, \hat{x}_0^\ell=\mathbf{x}_\theta^\ell(\mathbf{x}_t)\right).$$ The paper states that this approximation is only valid if the denoiser marginals are independent, which is not true for language. Tokens exhibit strong long-range dependencies, and the mismatch is particularly severe when masking ratios are high and visible context is sparse (Lemercier et al., 22 May 2026).

This produces the quality–throughput trade-off emphasized in the paper. If an MDLM samples many tokens in parallel, throughput improves, but the lack of explicit joint modeling over masked positions leads to incoherent token choices. If the model instead decodes conservatively, quality improves but the cost of iterative token-space denoising increases. DiLaDiff is designed to alter this trade-off by inserting a continuous latent variable that carries sentence-level semantics and token correlations, so that the discrete decoder is no longer forced to infer all inter-token dependencies from a partially masked sequence alone.

A common misconception is to treat DiLaDiff as merely a faster sampler for an unchanged MDLM. The paper’s actual claim is stronger and more specific: the latent variable is introduced to repair the independence failure of masked diffusion LLMs, and the speedup arises because the latent plan reduces the burden on the discrete reverse process rather than because token-space denoising itself becomes jointly exact.

2. Latent-augmented architecture

DiLaDiff has three components. The first is a continuous latent space learned by an auto-encoder fine-tuned from an existing masked diffusion LLM. The second is a continuous latent diffusion model that learns the prior over the encoder distribution. The third is a consistency model that distills the latent diffusion teacher into a few-step latent generator (Lemercier et al., 22 May 2026).

The auto-encoder introduces a latent code $\mathbf{z}$ intended to encode global semantics and token correlations. Its training objective combines masked reconstruction and latent conditioning: $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$ The paper stresses that this latent space is not learned from scratch using reconstruction alone. It is regularized by coordinate-wise normalization, feature masking/noising, latent masking/noising, occasional replacement of the latent with Gaussian noise, and decoder initialization from a pre-trained masked diffusion decoder. This combination is presented as the mechanism that makes the latent both semantic and diffusion-friendly.

Conditioning the decoder on the latent changes the denoiser from a purely token-space object into a latent-guided one: $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$ with reverse posterior

$$q_{s\mid t}^{\theta}(\mathbf{x}_s\mid \mathbf{x}_t) := \int \left( \prod_\ell q_{s\mid t}\!\left(\cdot \mid \mathbf{x}_t,\mathbf{x}_0^\ell=\mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z})\right) \right) p(\mathbf{z})\, d\mathbf{z}.$$

The paper’s interpretation is that conditional on the latent, token-level reverse transitions become substantially more coherent because $\mathbf{z}$ carries the dependencies that the original factorized reverse process omits.

The latent prior is then modeled by a continuous diffusion model. The latent denoiser $\psi$ is trained with

$$\mathcal{L}_\mathrm{LDM}^{\gamma} = \mathbb{E}_{\mathbf{z}\sim \mathcal{E}(\cdot)} \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{z}_t \sim \tilde q_t(\mathbf{z}_t \mid \mathbf{z})} } \left\| \mathbf{z}_\psi(\mathbf{z}_t,t) - \mathbf{z} \right\|_2^2,$$

under the variance-preserving process

$$\mathbf{z}_t = \alpha_t \mathbf{z} + \sigma_t \boldsymbol{\epsilon}, \qquad \boldsymbol{\epsilon}\sim \mathcal{N}(0,\mathbf{I}),$$

with

$$\mathrm{logSNR}(t) = -d \log \tan(\pi t/2), \qquad \sigma^2(t)=\mathrm{sigmoid}(-\mathrm{logSNR}(t)), \qquad \alpha^2(t)=\mathrm{sigmoid}(\mathrm{logSNR}(t)).$$

The defining DiLaDiff step is the final distillation. The paper uses MeanFlow to distill the latent diffusion teacher into a few-step student $\mathbf{z}$0, which learns the average velocity along the ODE trajectory: $\mathbf{z}$1 Training minimizes

$\mathbf{z}$2

with target

$\mathbf{z}$3

For $\mathbf{z}$4, the paper notes that the method reduces to standard flow matching.

3. Training pipeline and decoding procedure

The training pipeline has four stages. First, an MDLM baseline is trained on OpenWebText. Second, an auto-encoder is fine-tuned from that baseline: BERT hidden states are used as encoder inputs, the latent is learned via cross-attention, cross-attention layers are injected into the MDLM decoder, the decoder embedding table is frozen for stability, and the decoder is initialized from the MDLM checkpoint. Third, with the auto-encoder frozen, a continuous latent denoiser is trained on encoded latents. Fourth, the latent diffusion teacher is frozen and self-distilled into the MeanFlow student $\mathbf{z}$5, yielding DiLaDiff (Lemercier et al., 22 May 2026).

Inference differs sharply from standard masked diffusion decoding. In an MDLM, generation begins from all $\mathbf{z}$6 tokens and proceeds by repeated token prediction and unmasking, using only the token-space denoiser. In DiLaDiff, inference first generates a latent in a few steps using the distilled latent model, then feeds that latent into the masked decoder, and finally performs token-space unmasking conditioned on the latent. The paper characterizes this as adding a semantic latent plan before discrete decoding.

This division of labor is central. The latent generator operates in a compressed continuous space and avoids vocabulary projection and categorical sampling overhead. The discrete decoder remains the same masked diffusion decoder conditioned on the latent. DiLaDiff therefore does not eliminate token-space diffusion; it changes the conditioning structure so that fewer or more aggressive discrete decoding steps are viable without the same loss of coherence.

A second common misconception is that distillation replaces the latent diffusion model with a direct discrete generator. The paper does not make that claim. The distilled component is specifically the latent generative model; the token-space decoder is retained.

4. Reported empirical performance

The principal quantitative comparison is against the MDLM baseline on OpenWebText. The paper reports that MDLM attains GenPPL around $\mathbf{z}$7 and MAUVE around $\mathbf{z}$8. LaDiff with $\mathbf{z}$9 achieves GenPPL $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$0, MAUVE $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$1, about $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$2 latent diffusion overhead, about $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$3 relative GenPPL improvement over MDLM, and about $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$4 relative MAUVE improvement. LaDiff with $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$5 still outperforms MDLM with far fewer discrete decoding steps and gives around $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$6 wall-time acceleration, although latent diffusion overhead becomes significant if not distilled. DiLaDiff with $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$7 reduces latent overhead to about $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$8, makes latent generation negligible relative to discrete decoding, keeps MAUVE close to the teacher, and improves GenPPL and MAUVE over the masked diffusion baseline (Lemercier et al., 22 May 2026).

The paper repeatedly emphasizes that DiLaDiff comes close to the LaDiff teacher using only $$\mathcal{L}_\mathrm{AE-MDLM}^{\phi,\theta} = \mathbb{E}_{\substack{t \sim \mathcal{U}[0,1]\ \mathbf{x}_t \sim q_t(\mathbf{x}_t \mid \mathbf{x})}} \frac{\dot{\alpha}_t}{1-\alpha_t} \mathbb{E}_{\mathbf{z} \sim \mathcal{E}_\phi(\mathbf{z}\mid \mathbf{x})} \sum_\ell \log \langle \mathbf{x}_\theta^\ell(\mathbf{x}_t,\mathbf{z}), \mathbf{x}^\ell \rangle.$$9 latent steps instead of $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$0. That is the central efficiency claim: the semantic benefits of the latent prior are largely retained, while the expensive continuous diffusion trajectory is compressed into a few-step student.

The throughput argument is also architectural rather than merely benchmark-specific. Because the latent model runs in a compressed space and does not incur the full cost of token-space vocabulary projection and categorical sampling, the added latent stage is comparatively cheap. The reported throughput table is summarized in the paper by noting that, at batch size $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$1, LaDiff with $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$2 has a large overhead, whereas DiLaDiff with $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$3 reduces that overhead to about $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$4.

These results support a specific interpretation. The gain is not only that continuous diffusion can be accelerated by distillation; it is that latent-guided generation changes the operating point of discrete diffusion language modeling, allowing fewer token-space steps without the same degradation that ordinarily accompanies aggressive parallel decoding.

5. Ablations, latent-space properties, and decoding behavior

Several ablations define the method’s empirical profile. The paper finds that larger latent spaces improve reconstruction, but too little regularization makes the latent hard to model with diffusion; the best balance is reported as around $$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$5 compression. Among regularization strategies, “LaDiff / strongest augmentation” performs best overall, “MildAug” is reasonable but worse, “SoftAug” and weak dropout can yield high reconstruction but poor generative quality, and increasing encoder or decoder depth does not help much (Lemercier et al., 22 May 2026).

The latent is analyzed semantically rather than merely as a compression bottleneck. Adding noise to the latent progressively changes decoded meaning, and sentences decoded from the same latent are more semantically similar than sentences from different latents. The paper therefore argues that the latent captures semantic structure rather than just reconstruction-friendly statistics.

A particularly important ablation concerns masking ratio. The auto-encoder is reported to be much more robust than plain MDLM at high masking ratios, which directly supports the motivating claim that the latent provides information when token context is scarce. Temperature and confidence-based decoding experiments point in the same direction: the latent-guided model is more robust under temperature reduction and confidence-based token selection than plain MDLM, while MDLM tends to degenerate into repetitive high-confidence tokens.

The distillation stage is also ablated. MeanFlow and TVM show no major difference, but MeanFlow is adopted as the preferred distillation method. An extra-head self-conditioning design performs worse than the simpler extra-forward-pass self-conditioning. Taken together, these findings suggest that the latent prior is the primary representational contribution, while distillation is the mechanism that makes that prior practical at inference time.

6. Nomenclature and disambiguation

The label “DiLaDiff” is not unique across recent arXiv usage. In language modeling, it refers to “Distilled Latent-Augmented Diffusion for Language Modeling,” the method described above (Lemercier et al., 22 May 2026). In binary image comparison, however, the same string is used for the “dilated symmetric difference,” a morphological operator defined by

$$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$6

whose purpose is to suppress small misregistration errors while detecting genuine binary shape differences (Urieli, 30 May 2026).

A second source of confusion is “DLADiff,” which is a different method entirely. That term denotes “Dual-Layer Anti-Diffusion,” a dual-layer defense framework for facial privacy against fine-tuning-based and zero-shot customization of diffusion models, rather than a LLM (Jia et al., 25 Nov 2025). Closely related spellings in the diffusion-transformer literature also refer to different constructs, such as “$$q^\theta(\mathbf{x}_0 \mid \mathbf{x}_t) := \int \delta\!\left(\mathbf{x}_0 = \mathbf{x}_\theta(\mathbf{x}_t,\mathbf{z})\right)p(\mathbf{z})\,d\mathbf{z},$$7-DiT” for training-free acceleration of diffusion transformers and “DyDiLA” for Dynamic Differential Linear Attention.

For scholarly use, the topic is therefore best identified by full title and arXiv identifier rather than by the bare acronym alone. In the present sense, DiLaDiff denotes the distilled form of LaDiff: a latent-guided diffusion LLM in which a semantic continuous latent is first learned, then endowed with a diffusion prior, and finally distilled into a few-step latent generator so that latent planning becomes negligible in cost relative to discrete decoding.