Cola DLM: Continuous Latent Diffusion Model

• The paper introduces Cola DLM, a model that decouples global semantic planning from local text generation by mapping sequences into continuous latent space using a VAE and a diffusion prior.
• It employs a two-stage design where a text VAE captures local semantics and a block-causal diffusion prior orchestrates global coherence, enabling fast and modular generation.
• Empirical results show that Cola DLM outperforms traditional autoregressive models in tasks demanding global coherence and scales efficiently across high compute budgets.

Cola DLM (Continuous Latent Diffusion LLM) represents a paradigm shift in text generation architecture: it replaces the traditional strictly autoregressive token-by-token generation with a hierarchical process that separates global semantic planning from local textual realization. This is achieved via a two-stage design: a stable text-to-latent mapping realized by a variational autoencoder (VAE), and a block-causal diffusion model that acts as a global latent prior. As a result, Cola DLM enables non-autoregressive global information transport in continuous latent space, supporting efficient, semantically-rich, and modular generation, with extensibility to unified modeling across multiple continuous modalities (Guo et al., 7 May 2026).

1. Motivations and Generative Factorization

Conventional autoregressive (AR) LLMs decompose the data distribution using the left-to-right chain rule:

$$p_{\text{AR}}(x) = \prod_{i=1}^L p(x_i \mid x_{<i}),$$

which tightly couples semantic planning to the fixed token order and sequence length. This inherently sequential dynamic limits parallelism and binds global organization to local token emissions.

Cola DLM instead proposes a hierarchical factorization:

$$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$

where $z_0\in\mathbb{R}^d$ is a continuous latent capturing global semantics. The prior $p_\psi(z_0)$ allows the model to plan holistically in latent space before emitting text, introducing a non-AR inductive bias, enabling semantic compression, and modularizing prior modeling. From a Markov-path perspective, this factorization frames text generation as transporting a learned prior through a continuous latent Markov chain, decoupled from token order (Guo et al., 7 May 2026).

2. Text VAE Component: Encoder, Decoder, and Bottleneck

The first stage in Cola DLM is a text VAE mapping sequences to and from continuous latent variables:

• Encoder $q_\phi(z_0 \mid x)$: a strictly causal Transformer that maps a token sequence $x$ of length $T$ into a sequence of continuous vectors $z_0 \in \mathbb{R}^{T\times d}$.
• Decoder $p_\theta(x \mid z_0)$: another causal Transformer that reconstructs $x$ in a tokenwise fashion, conditioned on the entire $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$0.
• To combat posterior collapse and guarantee semantic information flow, a BERT-style masked-token loss $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$1 is added to the standard VAE ELBO:

$$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$2

By training with this bottleneck, the VAE ensures $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$3 captures compressible local semantics, leaving global structure for the subsequent diffusion prior. Experiments demonstrate that this stage yields smooth and robust x↔z₀ correspondence, with gradual semantic degradation under additive latent noise, indicating stable encoding (Guo et al., 7 May 2026).

3. Block-Causal Diffusion Prior and Generation Path

After the VAE, global semantic modeling is performed via a diffusion/flow model operating in latent space:

• Discrete-time: $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$4, and generation samples reversely via $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$5.
• Continuous-time (CNF): defines an ODE trajectory in latent space, $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$6.

The latent $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$7 is partitioned into $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$8 contiguous "blocks" $$p(x, z_0) = p_\theta(x\mid z_0)\,p_\psi(z_0), \quad p(x) = \int p_\theta(x\mid z_0)\,p_\psi(z_0)\,dz_0,$$9, and the global prior factorizes as

$z_0\in\mathbb{R}^d$0

so each block is conditioned only on previous ones. The model is trained via conditional Flow Matching; at inference, blocks are generated in parallel (intra-block) and sequentially (inter-block), with block count $z_0\in\mathbb{R}^d$1 offering orders-of-magnitude speedups compared to length-$z_0\in\mathbb{R}^d$2 autoregressive generation (Guo et al., 7 May 2026).

4. Decoding and Unified Markov Path Perspective

Once the global latent has been sampled via the diffusion prior, conditional decoding into text is performed:

• Conditional decoder $z_0\in\mathbb{R}^d$3 predicts the target tokens given the latent structure.
• If $z_0\in\mathbb{R}^d$4 is AR, generation remains sequential with respect to $z_0\in\mathbb{R}^d$5, but the global plan is decoupled from surface realization.

This generative path stands in contrast with AR, discrete diffusion, or embedding-space denoising: for Cola DLM, the generative path is "prior-transport" in latent space, with text emitted as the endpoint, strictly separating global semantics from local syntax and lexicalization (Guo et al., 7 May 2026).

5. Empirical Performance and Scaling Behavior

Cola DLM is evaluated against strictly matched baselines (∼2B-parameter AR and LLaDA, same token budgets, optimizer, sequence length) across a suite of benchmarks:

• Few-shot continuation: LAMBADA, SQuAD.
• Global semantics tasks: MMLU, RACE, OBQA, SIQA, Story Cloze, HellaSwag.

Scaling experiments ($z_0\in\mathbb{R}^d$6100–2000 EFLOPs) exhibit consistent improvements:

• Cola DLM matches or outperforms AR baselines on aggregate Task Avg at large compute budgets.
• On tasks demanding global coherence (MMLU, RACE, OBQA), Cola DLM's advantage is most pronounced at scale.
• For continuation, Cola DLM tracks AR, outperforming discrete-mask-diffusion (LLaDA) protocols (Guo et al., 7 May 2026).

6. Extensions and Multimodal Generalization

Cola DLM is architected for straightforward extensibility to unified, multimodal generation:

• By attaching parallel VAEs to other modalities (e.g., image, audio), mapping each to their latent $z_0\in\mathbb{R}^d$7, concatenating into a multimodal $z_0\in\mathbb{R}^d$8, and training a shared block-causal prior $z_0\in\mathbb{R}^d$9, the model supports joint text-image, image-to-text, and multi-modal continuation tasks.
• This modular design is validated by early prototypes: a single MMDiT prior can power text-to-image and image-to-text generation.
• Further open questions concern optimal rate–distortion trade-offs in latent compression, semantic alignment of block/de-noising schedules, and tighter likelihood calibration via reduced variational gap (Guo et al., 7 May 2026).

7. Significance and Theoretical Implications

Cola DLM establishes hierarchical continuous latent prior modeling as a principled alternative to token-level language modeling. Key properties include:

• Non-AR, semantically-aware generation reflecting actual model capability rather than likelihood-alone metrics.
• Direct support for semantic compression and modular multimodal extension.
• Scaling behavior indicating qualitative model improvement aligns with prior expressivity—contrasting with the stagnation or overfitting seen in standard AR as parameter counts increase. This architecture concretely realizes the separation between global structural planning and local realization, suggesting a general blueprint for flexible, scalable generative modeling across discrete and continuous modalities (Guo et al., 7 May 2026).