LDLM: Latent Diffusion for Language Generation

• LDLM is a class of language models that maps text into continuous latent spaces using diffusion or flow-based priors to encode global semantics.
• They separate semantic planning from local token realization, enabling efficient decoding through latent priors or hybrid strategies.
• Key architectural components include latent encoders, diffusion-based latent priors, and varied decoders that balance reconstruction and generative fidelity.

Latent Diffusion LLM (LDLM) denotes a family of language-generation architectures in which text is mapped into a continuous latent representation, a diffusion or flow-based prior is learned over that latent space, and denoised latents are decoded back into text. In the literature, the term covers both fully latent pipelines—where generation is framed as sampling a latent prior followed by conditional decoding—and hybrid systems in which latent diffusion serves as a semantic planner or guidance channel for an autoregressive or discrete diffusion decoder (Lovelace et al., 2022, Guo et al., 7 May 2026, Meshchaninov et al., 8 May 2026, Lovelace et al., 24 Feb 2026). Across these variants, the central design goal is to separate global semantic organization from local textual realization.

1. Definition and conceptual scope

LDLMs are motivated by a recurring claim in recent work: diffusion for language becomes more tractable when it is moved away from direct token-space denoising and into a continuous representation space. In this view, the latent variable is not merely an implementation detail but the primary object on which the generative prior is learned. "Latent Diffusion for Language Generation" introduced this pattern explicitly by compressing encoder states into a compact continuous latent and training diffusion there, then decoding with a pretrained seq2seq decoder (Lovelace et al., 2022). "Continuous Latent Diffusion LLM" makes the same commitment at a larger scale, defining language generation as a hierarchical latent-variable model in which a continuous prior over text latents is learned by a block-causal DiT and a decoder realizes those latents as text (Guo et al., 7 May 2026).

The scope of the term is not uniform. Some models use a heavily structured latent bottleneck, such as paragraph embeddings or continuation embeddings, while others keep a token-aligned latent sequence. TextLDM, for example, uses one latent vector per token rather than compressing sequence length, so its latent character is representational rather than spatially compressive (Jiang et al., 8 May 2026). By contrast, PLANNER diffuses over a small fixed set of paragraph embeddings, and STAR-LDM diffuses over a single 768-dimensional continuation embedding used for semantic planning (Zhang et al., 2023, Lovelace et al., 24 Feb 2026).

A separate but related usage of “latent” appears in discrete diffusion work on language. "Reasoning with Latent Tokens in Diffusion LLMs" uses “latent tokens” to mean still-unknown future token positions that remain active in the computation, not a continuous latent space (He et al., 3 Feb 2026). This distinction is important: latent diffusion in the canonical LDLM sense concerns continuous latent variables or latent trajectories, whereas latent-token mechanisms concern undecoded discrete positions inside a masked diffusion process.

2. Generative formulation and architectural decomposition

The canonical LDLM factorization is a hierarchical latent-variable model:

$$p(x,z_0)=p_\theta(x\mid z_0)\,p_\psi(z_0), \qquad p(x)=\int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0.$$

This form is stated explicitly in Cola DLM and underlies several other architectures in equivalent form (Guo et al., 7 May 2026). The latent prior $p_\psi(z_0)$ is learned by a diffusion or flow process, while $p_\theta(x\mid z_0)$ is a decoder conditioned on the denoised latent.

Most LDLMs instantiate the latent prior with a standard continuous noising relation,

$$z_t=\alpha_t z_0+\sigma_t\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$

or an equivalent flow-matching path (Zhang et al., 2023, Lovelace et al., 24 Feb 2026, Jiang et al., 8 May 2026). The decoder then receives either the clean latent $z_0$, a reconstructed memory derived from $z_0$, or a soft-prompt-like interface.

Three architectural components recur.

First, there is a latent encoder or text VAE. LD4LG compresses pretrained encoder states into a fixed-length latent tensor using a Perceiver Resampler (Lovelace et al., 2022). TextLDM uses a Transformer VAE with a per-token Gaussian latent sequence (Jiang et al., 8 May 2026). Cola DLM uses a causal Text VAE and treats the latent space as the site of semantic compression rather than token recovery (Guo et al., 7 May 2026).

Second, there is the latent prior. Some works use VP diffusion with $v$-prediction or $x_0$-prediction, while others use conditional flow matching or a continuous normalizing flow. TextLDM uses a standard DiT trained with conditional flow matching in the VAE latent space (Jiang et al., 8 May 2026). Cola DLM uses flow matching to learn what it calls latent prior transport rather than token-level observation recovery (Guo et al., 7 May 2026).

Third, there is the text decoder. This decoder may be parallel and non-autoregressive, as in TextLDM, or autoregressive, as in LD4LG, PLANNER, STAR-LDM, and CoDAR-style systems (Lovelace et al., 2022, Zhang et al., 2023, Lovelace et al., 24 Feb 2026, Shen et al., 3 Mar 2026). The persistence of autoregressive decoders is not incidental: several papers argue that latent diffusion is strong at global semantic search, while token realization remains easier for a conditional AR decoder.

3. Representative model families

The current literature contains several distinct LDLM families.

Model	Latent object	Decoding path
LD4LG (Lovelace et al., 2022)	Fixed-length continuous latent tensor from pretrained encoder states	Pretrained seq2seq decoder cross-attends to reconstructed latent memory
PLANNER (Zhang et al., 2023)	Paragraph embeddings $z=\{z^{(1)},\ldots,z^{(k)}\}$, typically $k=16, h=1024$	GPT-like autoregressive decoder realizes a paragraph-level plan
STAR-LDM (Lovelace et al., 24 Feb 2026)	768-dimensional Sentence-T5 XL continuation embedding, projected to 8 soft vectors	GPT-2 Large decodes after a latent “thinking” phase
TextLDM (Jiang et al., 8 May 2026)	Per-token Transformer-VAE latent sequence	Parallel non-autoregressive decoder
Cola DLM (Guo et al., 7 May 2026)	Hierarchical latent sequence from a Text VAE	Conditional decoder after block-causal DiT prior
DiLaDiff (Lemercier et al., 22 May 2026)	Continuous latent from compressed BERT features	Latent-guided masked diffusion decoder

One branch consists of pure or near-pure latent-prior models. LD4LG, TextLDM, and Cola DLM all treat the continuous latent prior as the main generative object, though they differ on whether sequence length is compressed and whether the prior is a DDPM-like diffusion or a flow-matching transport model (Lovelace et al., 2022, Jiang et al., 8 May 2026, Guo et al., 7 May 2026).

A second branch uses latent diffusion as a planner for a stronger discrete decoder. PLANNER constructs a paragraph-level semantic plan and then decodes with a GPT-like decoder (Zhang et al., 2023). STAR-LDM pauses before continuation generation, denoises a semantic continuation embedding, converts it into an 8-vector soft prompt, and then resumes autoregressive generation (Lovelace et al., 24 Feb 2026).

A third branch is hybrid continuous-discrete diffusion. DiLaDiff introduces a continuous latent prior over contextual text representations, then uses that latent to guide a masked diffusion decoder; consistency-style distillation further compresses the latent prior into a few-step generator (Lemercier et al., 22 May 2026). Closely related hybrid designs appear in Coevolutionary Continuous Discrete Diffusion and Latent Discrete Diffusion Models, both of which argue that continuous latents help with token correlation while discrete channels preserve trainability and decoding fidelity (Zhou et al., 3 Oct 2025, Shariatian et al., 20 Oct 2025).

4. Latent-space design and the decoding bottleneck

A central result across papers is that latent diffusion for text is limited less by the abstract idea of diffusion and more by the geometry of the latent space and the quality of the latent-to-token interface. "How to Train Your Latent Diffusion LLM Jointly With the Latent Space" states this as the central problem directly: the latent space must be easy both to denoise and to decode, and naive joint training produces a low-quality diffusion model (Meshchaninov et al., 8 May 2026). Its proposed training recipe combines an MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise; ablations show each component substantially impacts generation performance (Meshchaninov et al., 8 May 2026).

TextLDM reaches a closely related conclusion from a different direction. It reports that reconstruction fidelity alone is insufficient: across multiple VAE variants, reconstruction can remain near-perfect while downstream generation quality differs sharply, and REPA alignment to a frozen Qwen3-1.7B model is critical for obtaining latents “effective for conditional denoising” (Jiang et al., 8 May 2026). In its ablations, REPA raises WikiSource MAUVE from $p_\psi(z_0)$0 to $p_\psi(z_0)$1, Wikipedia BERTScore from $p_\psi(z_0)$2 to $p_\psi(z_0)$3, and TinyStories ROUGE-2 from $p_\psi(z_0)$4 to $p_\psi(z_0)$5 without materially changing reconstruction quality (Jiang et al., 8 May 2026).

A complementary diagnosis appears in CoDAR. That paper argues that the dominant failure mode of continuous diffusion language modeling is token recovery or rounding, not necessarily the quality of the continuous diffusion dynamics (Shen et al., 3 Mar 2026). Its theoretical decomposition attributes the gap between sequence-optimal decoding and pointwise rounding to conditional total correlation and a locality gap, and its token-recovery study is striking: at hidden dimension $p_\psi(z_0)$6, an autoregressive Transformer decoder recovers tokens at $p_\psi(z_0)$7 accuracy while a position-wise linear classifier reaches only $p_\psi(z_0)$8; at $p_\psi(z_0)$9, the AR decoder reaches $p_\theta(x\mid z_0)$0 while the linear classifier is still $p_\theta(x\mid z_0)$1 (Shen et al., 3 Mar 2026). CoDAR therefore reframes discretization as a sequence-transduction problem rather than a per-position projection problem.

DiLaDiff adds a further caveat. Its autoencoder ablations show that better reconstruction can be accompanied by much worse generation, which the paper interprets as evidence that latent spaces must be regularized for generatability rather than only for reconstruction (Lemercier et al., 22 May 2026). A similar lesson appears in "Towards Latent Diffusion Suitable For Text": once diffusion is moved to a continuous text representation, the learned forward process becomes a major design choice, and a data-adaptive multivariate forward process can substantially reduce the likelihood gap with autoregressive models of the same size while only achieving sample quality comparable to previous latent diffusion models (Midavaine et al., 7 Jan 2026).

5. Decoding, controllability, and efficiency

LDLMs differ markedly in how they turn denoised latents into text. Planner-decoder systems deliberately split semantic planning from textual realization. PLANNER uses latent semantic diffusion to generate paragraph embeddings and then a decoder to realize them as fluent text (Zhang et al., 2023). STAR-LDM similarly inserts a “Stop-Think-AutoRegress” phase in which a continuation embedding is denoised before GPT-2 Large resumes token generation; it also supports classifier-free prefix guidance and latent classifier guidance for sentiment and toxicity (Lovelace et al., 24 Feb 2026).

Other models treat the decoder itself as the critical interface. CoDAR’s contextual autoregressive decoder cross-attends to the denoised embedding sequence and exposes a decoder-temperature knob that controls the fluency-diversity frontier (Shen et al., 3 Mar 2026). This design makes discretization an explicit conditional sequence model rather than an afterthought.

Efficiency remains a persistent issue. DiLaDiff addresses it by distilling the latent prior with a MeanFlow-style consistency model: after distillation, a setting with $p_\theta(x\mid z_0)$2 and $p_\theta(x\mid z_0)$3 reduces latent overhead to $p_\theta(x\mid z_0)$4, which the paper describes as negligible relative to discrete decoding (Lemercier et al., 22 May 2026). The paper’s broader claim is that a continuous latent prior can improve both quality and throughput by letting the discrete decoder unmask many tokens in parallel more coherently (Lemercier et al., 22 May 2026).

The acceleration literature for masked diffusion LMs is not itself an LDLM literature, but it exposes deployment constraints that latent systems also inherit. "Accelerating Diffusion LLM Inference via Efficient KV Caching and Guided Diffusion" reports up to a $p_\theta(x\mid z_0)$5 end-to-end speedup through training-free FreeCache and Guided Diffusion on masked DLMs (Hu et al., 27 May 2025). "Efficient Diffusion LLMs via Temporal-Spatial Parallel Decoding and Confidence Extrapolation" goes further by freezing tokens whose denoising trajectories appear converged, reaching a $p_\theta(x\mid z_0)$6 speedup at 1024 generated tokens while preserving accuracy on GSM8K (Li et al., 29 May 2026). These results are reported for discrete masked diffusion backbones rather than canonical latent-prior LDLMs, but a plausible implication is that analogous cache reuse, early freezing, and trajectory-aware stopping criteria should also matter for latent denoising trajectories.

6. Interpretability, hybridization, and open problems

Recent work suggests that diffusion LLMs may internally construct meaningful latent control variables even when these are not explicit nodes in the generative graph. "Subliminal Clocks: Latent Time Modelling in Diffusion LLMs" shows that in timestep-unconditioned masked DLMs, a denoising-progress signal $p_\theta(x\mid z_0)$7 emerges in the residual stream, is decodable across layers, can be causally steered, and occupies a low-dimensional manifold-like geometry (Rulli et al., 2 Jul 2026). The paper does not formulate $p_\theta(x\mid z_0)$8 as an explicit latent variable in a generative model, but it treats denoising progress as a model-internal state variable that is encoded, manipulated, corrected, and geometrically organized (Rulli et al., 2 Jul 2026). This suggests that explicit latent priors in LDLMs may coexist with emergent internal latents learned by the denoiser itself.

Hybridization is therefore becoming a defining theme rather than an exception. CCDD argues that continuous diffusion is more expressive than discrete diffusion but hard to decode, and proposes a joint continuous-discrete diffusion process in which a continuous representation space and a discrete token space are denoised together (Zhou et al., 3 Oct 2025). LDDM makes a closely related argument from the factorization bottleneck of masked discrete reverse transitions, introducing both fully joint and sequential latent-discrete variants (Shariatian et al., 20 Oct 2025). DiLaDiff, likewise, is best understood as a cascaded latent-guided discrete decoder rather than a pure latent-only generator (Lemercier et al., 22 May 2026).

Several unresolved issues recur across the literature. One is the proper latent granularity: TextLDM uses one latent per token, Cola DLM studies patch sizes $p_\theta(x\mid z_0)$9 and $$z_t=\alpha_t z_0+\sigma_t\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$0, PLANNER uses paragraph embeddings, and STAR-LDM uses a single continuation embedding (Jiang et al., 8 May 2026, Guo et al., 7 May 2026, Zhang et al., 2023, Lovelace et al., 24 Feb 2026). Another is the balance between generation quality and likelihood. Cola DLM argues that for this class of models, likelihood and perplexity can be structurally misaligned with generation quality because generation depends on reaching decoder-valid latent regions, whereas likelihood demands precise density calibration around the gold posterior neighborhood (Guo et al., 7 May 2026). A third is whether language is ultimately better served by pure latent priors, hybrid latent-discrete systems, or planner-decoder decompositions.

The field’s most stable conclusion is narrower but consequential: diffusion for language becomes substantially more viable when latent structure is treated as a first-class modeling object rather than as a by-product of token denoising. Whether that object is a compact paragraph plan, a token-aligned VAE latent sequence, a semantic continuation embedding, or a hybrid latent channel coupled to discrete decoding, the defining LDLM hypothesis is that global structure is easier to model in continuous latent space than at the level of direct token recovery (Lovelace et al., 2022, Meshchaninov et al., 8 May 2026, Guo et al., 7 May 2026).