Coevolutionary Continuous Discrete Diffusion

• CCDD is a generative modeling paradigm that jointly evolves continuous and discrete representations via a coupled diffusion process, ensuring both semantic expressivity and reliable token decoding.
• The framework employs stochastic differential equations alongside continuous-time Markov chains to enable tractable denoising and robust training across modalities.
• Empirical results indicate that CCDD reduces perplexity in language modeling tasks by fusing rich latent semantics with stable discrete supervision.

Coevolutionary Continuous Discrete Diffusion (CCDD) defines a generative modeling paradigm in which both continuous and discrete modalities are jointly corrupted and denoised over time via a coupled diffusion process. The approach bridges the high semantic expressivity and reasoning capacity of continuous representations with the robust trainability and discrete supervision of categorical token spaces. By evolving both modalities simultaneously, CCDD enables diffusion models to benefit from rich latent semantics, stable discrete anchors, and tractable generative sampling, overcoming the limits exhibited by purely continuous or discrete diffusion models. It provides a rigorous framework for language modeling and other structured-data tasks, supported by novel theoretical insights, advanced architectures, and strong empirical evidence from state-of-the-art experiments.

1. Theoretical Expressivity and Foundations

CCDD is predicated on the theoretical observation that continuous diffusion processes are strictly more expressive than discrete diffusion models and architectures based on looped transformers. The continuous process, governed by stochastic differential equations (SDEs) or probability flow ODEs such as

$z_t = \alpha_t z_0 + \sigma_t \varepsilon$

(with $\alpha_t, \sigma_t$ determined by the noise schedule), yields absolutely continuous marginals in latent space for any $t > 0$. In contrast, discrete diffusion produces finite, atomic distributions even when tokens are embedded in a continuous space. Formally, for any model parameter $\theta$, if $\mathcal{F}_{\text{cont}}(\theta)$ defines the family of trajectory laws for continuous diffusion, and $\tilde{\mathcal{F}}_{\text{disc}}(\theta)$ the family for discrete, one has

$$\tilde{\mathcal{F}}_{\text{disc}}(\theta) \subsetneq \mathcal{F}_{\text{cont}}(\theta).$$

Moreover, the continuous process can simulate rollouts of looped transformers by aligning the reverse probability flow ODE with a discretized Euler update, yielding equivalence with transformer updates for particular schedule choices. Thus, CCDD leverages both modalities to maximize trajectory-level expressiveness (Zhou et al., 3 Oct 2025).

2. Joint Diffusion Process: Modeling and Architecture

The CCDD framework jointly evolves a discrete token space (e.g., categorical vocabulary) and a continuous representation space (e.g., contextual embeddings). The forward (noising) process is factorized so that:

• The discrete part is typically modeled by a continuous-time Markov chain (CTMC) over the vocabulary, governed by a rate matrix $Q_t$, and evolves via jumps or masking.
• The continuous part evolves according to an SDE, often in the shared embedding space, with noise injection governed by a standard Gaussian or prescribed variance schedule.

Mathematically, if $x_t$ denotes the sequence of discrete tokens and $z_t$ the corresponding continuous vectors, the joint process is described by coupled update rules such as:

$$q(x_t, z_t \mid x_0, z_0) = q(x_t \mid x_0) q(z_t \mid x_t, z_0)$$

Sampling and denoising are performed using a single model $f_\theta(\cdot, t)$, which has two heads: one predicting the “clean” continuous representation (regression loss on $z_t$), and another outputting discrete token logits (cross-entropy loss). Architectures are based on advanced transformer models (e.g., DiT, multimodal DiT, mixture-of-experts MoE), augmented to handle multimodal streams, conditional masking, and cross-modal conditioning (Zhou et al., 3 Oct 2025).

3. Overcoming Trainability and Decoding Challenges

A core challenge for continuous diffusion in language modeling is decoding from continuous representations to valid discrete tokens, owing to the high-dimensional and non-quantized latent space. This leads to ambiguity and degraded performance when mapping Gaussian outputs to vocabulary indices.

CCDD resolves this by co-evolving explicit token trajectories, which serve as anchors for mapping the continuous latent. During denoising:

• The discrete branch predicts token labels, stabilizing the generation and ensuring alignment with the vocabulary.
• The continuous branch provides semantic guidance to discrete decoding, enabling soft reasoning and supporting richer context modeling.

Techniques such as asynchronous noise schedules (slower decay in continuous space, faster in discrete) are employed so that high-level semantic information is retained when the discrete signal becomes noisy, facilitating effective co-conditioning during inference.

Furthermore, classifier-free guidance (CFG) and self-conditioning are incorporated to blend conditional/unconditional generations. The reverse updates are factored for each modality but conditioned on the other, enabling flexible yet coupled generative trajectories (Zhou et al., 3 Oct 2025).

4. Training, Objectives, and Analytical Tools

CCDD training combines regression losses on continuous (“$\epsilon$-prediction” for $z_t$) and cross-entropy losses on discrete tokens. Joint optimization ensures that both branches provide complementary gradients. Theoretical results from stochastic integral frameworks confirm that error can be decomposed into components such as:

• Truncation error: $\exp(-\rho T)\log |\mathcal{S}|$ (decay over finite-time stationary approximation)
• Approximation error: score function estimation error, $ϵ$
• Discretization error: due to time-step $\kappa$, scaling as $(\overline{D})^2 \kappa T$ These error bounds, established for both pure discrete (Ren et al., 4 Oct 2024) and coevolutionary models, provide rigorous guidance for selecting training parameters and designing sampling algorithms (e.g., $\tau$-leaping for discrete, Euler-Maruyama for continuous components), ensuring that the joint diffusion process converges in KL divergence to the desired generative distribution.

5. Empirical Results and Performance

CCDD achieves substantial empirical improvements in language modeling tasks over both discrete-only and continuous-only baselines. On benchmarks including LM1B and OpenWebText:

• CCDD models, such as CCDD-MDiT with Qwen3 embeddings, achieve validation perplexities of $≤29.22$, versus $\sim 39.17$ for Masked Diffusion LLMs (MDLM) (Zhou et al., 3 Oct 2025).
• When trained with contextualized embeddings (e.g., RoBERTa, Qwen3-Embedding), performance improves in both MSE (for continuous regression) and token-level accuracy (cross-entropy), with significant reductions in token perplexity.
• Inference-time guidance and ablation studies demonstrate the impact of cross-modal architectures and embedding layer choice. These results demonstrate that the fusion of discrete supervision and continuous reasoning in CCDD delivers superior generative quality, parallel decoding, and improved self-correction compared to previous architectures.

6. Related Methodologies and Comparative Approaches

CCDD sits at the intersection of several methodological advances:

• Lifting discrete data into continuous space: As in CDCD (Dieleman et al., 2022), which uses embedding-based Gaussian diffusion for categorical data, jointly parameterized by cross-entropy loss.
• Discrete-state continuous-time diffusion: CTMC-based models for graphs (Xu et al., 19 May 2024), music, and VQ-image generation (Sun et al., 2022, Zhao et al., 6 Feb 2024), which preserve categorical structure while enabling efficient, flexible sampling.
• Continuously augmented discrete diffusion: CADD (Zheng et al., 1 Oct 2025) introduces continuous latents as semantic hints augmenting mask-based discrete “voids,” providing soft guidance to alleviate mode collapse and information loss.
• Latent variable augmentation: DisCo-Diff (Xu et al., 3 Jul 2024) attaches discrete latents to continuous diffusion trajectories for global, mode-separating structure, simplifying the generative ODE and improving sample quality.
• Unified stochastic integration: Analytical tools leveraging Poisson random measures and change-of-measure theorems provide rigorous error decomposition for general (co)evolutionary processes (Ren et al., 4 Oct 2024).

Each of these lines contributes crucial components: continuous expressivity, discrete supervision, multimodal inference, and unified theoretical analysis, all synthesized in the CCDD framework.

7. Outlook and Research Directions

CCDD highlights several promising research avenues:

• Latent Reasoning and Chain-of-Thought: Leveraging continuous latent spaces for iterative latent planning, parallel exploration, and chain-of-thought reasoning, as discussed in theoretical analysis of continuous versus looped transformer expressivity (Zhou et al., 3 Oct 2025).
• Multimodal and Compositional Generation: Extending joint processes to multimodal domains (text-vision, text-code) where coupled state spaces and cross-guidance improve semantic richness and model robustness.
• Continual Generative Learning: Incorporating knowledge consistency hierarchies (Liu et al., 17 May 2025) to evolve models in streaming/continual settings without catastrophic forgetting, potentially using cross-modal analogs of inter-task, unconditional, and label consistency.
• Sample-Efficiency and Numerical Optimization: Exploiting recent stochastic analysis (Ren et al., 4 Oct 2024) for constructing optimal $\tau$-leaping schedules and synthesizing hybrid inference methods for high-dimensional discrete–continuous state spaces.
• Practical Applications: Robust, parallel text generation; high-precision semantic infilling; program synthesis; and molecular graph generation.

In summary, Coevolutionary Continuous Discrete Diffusion provides a unified, expressive, and efficient paradigm that integrates the semantic strengths of continuous latent representations with the robust, interpretable, and stable supervision of discrete generative models. Its rigorous theoretical underpinnings, advanced architectural designs, and strong empirical results establish a foundation for the next generation of multimodal generative systems.