Continuous Diffusion for Discrete Text

• Continuous diffusion for discrete text is a method that adapts continuous-time diffusion processes to model sequences of discrete tokens via embedding spaces and statistical manifolds.
• It employs a forward process with noise injection through SDEs and a learned reverse denoising network to recover token information from continuous representations.
• Hybrid strategies integrate discrete transitions with continuous flows, improving sampling efficiency, alignment of token boundaries, and overall generation quality.

Continuous diffusion for discrete text refers to the adaptation of continuous-time generative diffusion models—originally developed for image and signal processing—to the modeling and generation of sequences comprised of discrete symbols, such as natural language. Rather than treating text tokens as atomic categorical variables and applying only discrete Markov transitions, these methods propose to represent discrete text in an appropriate continuous space—embedding vectors, the statistical manifold of categorical distributions, or other geometrically structured domains—and then define or learn a continuous-time (or continuous-valued) diffusion process. The approach supports parallel, non-autoregressive decoding and enables the use of advanced tools such as ODE/SDE solvers and flow matching, but also requires careful handling given the mismatch between the continuous generative path and the discrete target structure.

1. Foundational Frameworks and Methodologies

Continuous diffusion models for discrete text begin by projecting the discrete token sequence $x = (x_1,\dots,x_n)$ into a continuous space:

• Embedding space: Each token is mapped via a (possibly learned) embedding matrix $E: \mathcal{V} \to \mathbb{R}^d$, yielding $z_0 = (E[x_1],\dots,E[x_n])$ (Strudel et al., 2022, Yang et al., 18 May 2026, Chen et al., 13 Apr 2026).
• Statistical manifold (simplex): Tokens are represented as one-hot vectors, and the simplex of all categorical distributions (probability vectors) is endowed with Riemannian structure as in RDLM (Jo et al., 17 Feb 2025).
• Latent geometry: Some approaches, such as NFDM (Midavaine et al., 7 Jan 2026), learn flexible latent spaces adapted to text sequence statistics.
• Hybrid manifolds: Methods like CANDI (Pynadath et al., 26 Oct 2025) or CCDD (Zhou et al., 3 Oct 2025) maintain both discrete token and continuous latent representations, coupling them through joint diffusion.

The forward process is typically a continuous-time SDE (e.g., "variance preserving" or "VE") that injects Gaussian noise into these representations:

$$dz_t = -\frac{1}{2}\beta(t)z_t\,dt + \sqrt{\beta(t)}\,dW_t$$

with closed-form Gaussian marginals $$q(z_t|z_0) = \mathcal{N}(\alpha(t)z_0, \sigma^2(t)I)$$ for appropriate schedule parameters. In more geometric approaches, diffusion may be conducted on the statistical manifold of distributions via Brownian motion or geodesic flows (Jo et al., 17 Feb 2025). Several recent works further introduce per-token or adaptive noise schedules (e.g., NeoDiff (Li et al., 28 May 2025), LangFlow (Chen et al., 13 Apr 2026)) or decouple the continuous and discrete corruption processes (CANDI (Pynadath et al., 26 Oct 2025)).

The reverse (generative) process is defined via:

• A learned denoising network (often a bidirectional transformer) that either predicts the original representation or the "score function" (gradient of the log-density).
• A deterministic or stochastic reverse SDE/ODE for generation, using the learned score or conditional mean.
• Discretization of the continuous output back to tokens, usually via nearest-neighbor or dot-product similarity with the embedding matrix, or softmax over vocabularies.

Hybrid approaches combine continuous diffusion over embeddings with explicit discrete transitions—absorbing mask processes or structured categorical corruptions—and synchronize the generative steps (CANDI (Pynadath et al., 26 Oct 2025), CCDD (Zhou et al., 3 Oct 2025), NeoDiff (Li et al., 28 May 2025), CoM-DAD (Xu et al., 7 Jan 2026)).

2. Geometries and Theoretical Foundations

A crucial insight in extending diffusion to text is the recognition that the supporting space for discrete data (the probability simplex) admits smooth Riemannian geometry, enabling the definition of continuous-time flows that generalize standard discrete diffusion:

• Riemannian simplex flow: RDLM (Jo et al., 17 Feb 2025) establishes that discrete-step masked or uniform corruption is a special case of continuous geodesic interpolation on the simplex, parameterized by great-circle (slerp) paths.
• Mixtures of bridges: The forward process can be constructed as a mixture of terminal-point-conditioned bridges, with precise drift and diffusion coefficients defined by the geometry of categorical distributions.
• Dimension-splitting: For large vocabularies, the simplex is decomposed into the product of lower-dimensional submanifolds (Jo et al., 17 Feb 2025), facilitating smoother transitions and tractable score parameterization.
• Boundary conditioning: Boundary Conditional Diffusion (Gu et al., 2024) introduces boundary-aware forward processes, rescaling the diffusion trajectory to the true decision region associated with token reconstruction, thus aligning density contours with the discrete rounding map.

Another foundational result is that continuous diffusion is strictly more expressive than discrete diffusion or looped transformers at the path level: the law of a continuous SDE has absolutely continuous marginals in $\mathbb{R}^{L \times d}$, whereas any discrete process has support on a finite subset (Zhou et al., 3 Oct 2025).

3. Algorithmic Innovations and Model Variants

Multiple algorithmic and architectural strategies have been advanced to close the accuracy and efficiency gap between continuous diffusion and discrete/AR models:

• Self-conditioning: Feeding previous denoising outputs as auxiliary inputs enhances likelihood and sample quality, with especially pronounced gains in models such as LangFlow (Chen et al., 13 Apr 2026) and RePlaid (Yang et al., 18 May 2026).
• Flow matching and ODE-ELBO: LangFlow (Chen et al., 13 Apr 2026) leverages Bregman-divergence flow matching, deriving a principled ODE-based negative log-likelihood bound for continuous flows that improves training stability and enables accurate NLP perplexity estimation.
• Poisson diffusion and intrinsic time: NeoDiff (Li et al., 28 May 2025) introduces fine-grained, token-specific intrinsic time schedules via Poisson jump processes, enhancing semantic expressivity over globally uniform diffusion.
• Soft absorbing states: DiffuSeq-v2 (Gong et al., 2023) combines continuous Gaussian corruption with "soft masking," facilitating the learning of hybrid denoising and accelerating sampling via efficient ODE solvers.
• Hybrid discrete-continuous corruption: CANDI (Pynadath et al., 26 Oct 2025) decouples discrete identity corruption (masking) from continuous rank degradation (Gaussian noise) to avoid temporal dissonance between conditional structure and geometry.

For accelerated sampling, many frameworks employ ODE-based probability-flow solvers or multistep DPM-solver++ routines, reducing the number of required function evaluations by several orders of magnitude compared to vanilla ancestral/Markov chain sampling (Gong et al., 2023, Li et al., 28 May 2025).

4. Empirical Performance and Scaling

Recent works, notably LangFlow (Chen et al., 13 Apr 2026) and RePlaid (Yang et al., 18 May 2026), demonstrate that continuous diffusion models can now match or outperform discrete DLMs and autoregressive LMs on large-scale language modeling tasks:

• Perplexity: On OpenWebText (context=1024), RePlaid achieves a perplexity lower bound of 22.1, a new state-of-the-art among continuous DLMs (Yang et al., 18 May 2026). LangFlow attains 24.6, also surpassing numerous discrete baselines (Chen et al., 13 Apr 2026).
• Scaling laws: For a given non-embedding FLOP budget, RePlaid maintains a compute gap of only $20\times$ to AR models (vs. historical $64\times$), and achieves parameter economy over discrete Masked DLM and Duo baselines by over 1.8x (Yang et al., 18 May 2026).
• Sample quality and diversity: GlyphDiffusion (Li et al., 2023) and NeoDiff (Li et al., 28 May 2025) demonstrate improved BLEU/ROUGE and enhanced diversity metrics in paraphrasing, translation, and open-domain tasks, outperforming both AR and other diffusion-based approaches in specific settings.

Hybrid and boundary-conditioned models (CANDI (Pynadath et al., 26 Oct 2025), BCD (Gu et al., 2024), CoM-DAD (Xu et al., 7 Jan 2026)) further report consistent outperformance over classical mask-based and uniform discrete models, particularly at low-step sampling regimes and on conditional generation tasks.

5. Challenges, Limitations, and Structural Alignment

Despite recent advances, continuous diffusion for discrete text faces nontrivial challenges:

• Discrete-continuous mismatch: Naive continuous embedding-space diffusion erases token-level identity at high noise rates, especially for large vocabularies—causing a collapse of conditional information before neighborhood structure can be exploited (Pynadath et al., 26 Oct 2025, Jin et al., 27 Dec 2025).
• Marginal training limitations: Per-token marginal losses cannot enforce proper multi-token dependencies, permitting "marginal traps" where spurious combinations dominate (Jin et al., 27 Dec 2025).
• Boundary misalignment: Gaussian contours may not coincide with the true boundaries of discrete decision regions, resulting in suboptimal rounding (Gu et al., 2024).
• Noise scheduling: Uniform or non-adaptive schedules waste time-steps at low signal-to-noise or fail to allocate denoising effort proportionally to information loss (Chen et al., 13 Apr 2026).

To address these, structural alignment principles have been articulated:

• Hybrid discrete-continuous processes: Decouple the corruption and denoising rates for identity and geometry to preserve semantic recoverability (Pynadath et al., 26 Oct 2025, Zhou et al., 3 Oct 2025).
• Information-uniform scheduling: Sample and allocate timesteps proportional to the marginal entropy decay (information loss) (Chen et al., 13 Apr 2026).
• Boundary conditioning: Force the diffusion trajectory to respect the true rounding boundary, improving precision and stability (Gu et al., 2024).
• Context-weighted training: Weight loss terms by token information content or local recoverability (Jin et al., 27 Dec 2025).

6. Unified and Hybrid Models

Recent work emphasizes the unification of continuous and discrete approaches as a pathway to principled and effective text diffusion:

• Analytical unification: USD3 (Zhao et al., 2024) shows that continuous-time categorical diffusion (via CTMC) subsumes discrete-time versions as a special case, and that backward sampling and ELBO losses have closed-form, efficient expressions that are consistent across settings.
• Coupled manifold diffusion: CoM-DAD (Xu et al., 7 Jan 2026) and related hierarchical dual-process models decouple high-level semantic planning (continuous diffusion on latent manifolds) from low-level token selection (absorbing-state discrete diffusion), facilitating multimodal generation and alignment.

These architectures achieve robust performance across information retrieval, summarization, translation, paraphrasing, and open-domain dialogue benchmarks.

7. Empirical Highlights and Future Directions

Continuous diffusion models now match discrete diffusion and AR models in both perplexity and sample quality on standard benchmarks, including OpenWebText, LM1B, and various conditional generation tasks (Yang et al., 18 May 2026, Chen et al., 13 Apr 2026, Li et al., 28 May 2025). Scaling laws validate the approach's parameter and compute efficiency. However, challenges remain in fully aligning continuous generative flow with discrete target structure, preventing mode collapse and capturing complex dependencies.

Further research focuses on:

• Improved hybridization of discrete and continuous representations
• Advanced adaptive noise scheduling and per-token control
• Closed-form or simulation-free techniques for high-dimensional manifolds (Jo et al., 17 Feb 2025)
• Incorporating classifier-based and energy-based controls for conditional generation (Pynadath et al., 26 Oct 2025)
• Integration with large pretrained LLMs to harness emergent reasoning capabilities

The trajectory of the field suggests that principled continuous diffusion, with proper structural alignment and hybridization, will continue to advance as a competitive alternative for modeling discrete text sequences.