CANDI: Hybrid Discrete-Continuous Diffusion Models (2510.22510v2)

Abstract: While continuous diffusion has shown remarkable success in continuous domains such as image generation, its direct application to discrete data has underperformed compared to purely discrete formulations. This gap is counterintuitive, given that continuous diffusion learns score functions that enable joint evolution across multiple positions. To understand this gap, we introduce token identifiability as an analytical framework for understanding how Gaussian noise corrupts discrete data through two mechanisms: discrete identity corruption and continuous rank degradation. We reveal that these mechanisms scale differently with vocabulary size, creating a temporal dissonance: at noise levels where discrete corruption preserves enough structure for conditional learning, continuous denoising is trivial; at noise levels where continuous denoising is meaningful, discrete corruption destroys nearly all conditional structure. To solve this, we propose CANDI (Continuous ANd DIscrete diffusion), a hybrid framework that decouples discrete and continuous corruption, enabling simultaneous learning of both conditional structure and continuous geometry. We empirically validate the temporal dissonance phenomenon and demonstrate that CANDI successfully avoids it. This unlocks the benefits of continuous diffusion for discrete spaces: on controlled generation, CANDI enables classifier-based guidance with off-the-shelf classifiers through simple gradient addition; on text generation, CANDI outperforms masked diffusion at low NFE, demonstrating the value of learning continuous gradients for discrete spaces. We include the code on the project page available here: https://patrickpynadath1.github.io/candi-lander

• The paper presents a novel hybrid diffusion model that decouples discrete and continuous corruption to preserve token identifiability in large-vocabulary domains.
• It introduces a theoretical token identifiability framework, revealing that discrete identity corruption scales exponentially with vocabulary size, causing temporal dissonance.
• Empirical results on datasets like Text8 and OpenWebText validate that CANDI achieves superior low-NFE performance and enables controllable generation using flexible guidance.

CANDI: Hybrid Discrete-Continuous Diffusion Models

Introduction and Motivation

The CANDI framework addresses a fundamental limitation in applying continuous diffusion models to discrete data, such as text. While continuous diffusion models have demonstrated strong performance in continuous domains (e.g., images), their direct application to discrete domains has consistently underperformed compared to purely discrete diffusion models. This is counterintuitive, as continuous diffusion models, in principle, can leverage joint updates and score-based learning, which should be advantageous for parallel generation and efficient inference.

The paper introduces a theoretical framework—token identifiability—to analyze the interaction between Gaussian noise and discrete data. It identifies two distinct mechanisms by which noise corrupts discrete data: discrete identity corruption (loss of token identity under argmax) and continuous rank degradation (loss of relative rank among tokens). Crucially, these mechanisms scale differently with vocabulary size, leading to a temporal dissonance: at noise levels where discrete structure is preserved, continuous denoising is trivial; at noise levels where continuous denoising is meaningful, discrete structure is lost. This mismatch explains the empirical failure of continuous diffusion in large-vocabulary discrete domains.

Figure 1: A visual comparison of discrete, continuous, and hybrid diffusion. CANDI leverages both discrete conditional structure and joint updates.

Token Identifiability and Temporal Dissonance

The core insight is that discrete identity corruption (the probability that the correct token is no longer the argmax after noise) increases exponentially with vocabulary size, while continuous rank degradation (the expected fraction of incorrect tokens ranked above the correct one) is independent of vocabulary size. This leads to a regime where, for large vocabularies, the model cannot simultaneously learn conditional structure and continuous score functions.

Figure 2: Visualization of discrete and continuous corruption mechanisms and their asymmetric scaling with vocabulary size $|V|$.

Figure 3: Plots of $\rho(\sigma)$ (discrete corruption) and $r(\sigma)$ (continuous degradation) for varying $|V|$, illustrating their divergence as vocabulary size increases.

Empirical results confirm the analytical predictions, showing that for small vocabularies, both mechanisms degrade in tandem, but for large vocabularies, discrete identity is lost much earlier than continuous signal.

Figure 4: Empirical results closely match analytical expressions for both discrete and continuous corruption.

The CANDI Framework

CANDI (Continuous ANd DIscrete diffusion) introduces a hybrid noising kernel that decouples discrete and continuous corruption. Specifically, it applies a position-wise masking process (as in masked diffusion) to preserve a subset of clean tokens, while applying Gaussian noise to the remaining positions. This design ensures that both discrete identity corruption and continuous rank degradation scale linearly with time, eliminating the problematic $|V|$-dependence.

Figure 5: Single-point evaluations for perplexity and entropy can be misleading; temperature sweeps are necessary for robust comparison.

Figure 6: Generative frontiers on Text8 (small vocab) and OWT (large vocab) for various methods. CANDI matches or exceeds MDLM performance and avoids the collapse seen in pure continuous diffusion on large vocabularies.

The forward process is defined as:

$X_t = X_0 \odot M_t + \tilde{X}_t \odot (1 - M_t)$

where $M_t$ is a Bernoulli mask with a linear schedule, and $\tilde{X}_t$ is a Gaussian-corrupted version of $X_0$. The reverse process combines discrete updates (for masked positions) and continuous ODE-based updates (for noisy positions), leveraging both conditional structure and continuous gradients.

Model Parameterization and Inference

The model is parameterized to predict $P_\theta(X_0 | X_t)$ and $\nabla \log p_t(X_t)$, enabling both discrete and continuous reverse transitions. The training objective is a reweighted cross-entropy loss, reflecting the proportion of corrupted positions.

For inference, CANDI provides both an exact algorithm (requiring full matrix operations) and an approximate algorithm (using a single-step Monte Carlo estimate and embedding lookups), the latter matching the efficiency of pure discrete methods without significant loss in generative quality.

Figure 7: The approximate inference algorithm is competitive with the exact algorithm, with no significant degradation in generative quality.

Empirical Results

Vocabulary Scaling and Temporal Dissonance

Experiments on Text8 (small vocabulary) and OpenWebText (large vocabulary) validate the temporal dissonance hypothesis. Pure continuous diffusion performs reasonably on small vocabularies but fails catastrophically on large ones. CANDI, by decoupling the two corruption mechanisms, maintains strong performance regardless of vocabulary size.

Low-NFE Generation and Efficiency

CANDI demonstrates superior generative quality at low numbers of function evaluations (NFE), outperforming masked diffusion (MDLM) and uniform diffusion (DUO) up to 64 NFE. This is attributed to the ability of continuous gradients to coordinate updates across many positions in parallel.

Figure 8: Entropy-perplexity frontiers for various NFE. CANDI outperforms MDLM up to 64 NFE and surpasses DUO as perplexity falls below 40.

Figure 9: Throughput comparison: autoregressive models (with KV caching) are faster at high NFE, but diffusion models (including CANDI) are more efficient at low NFE.

Controllable Generation

CANDI enables classifier-based guidance using off-the-shelf classifiers via simple gradient addition, without requiring diffusion-specific classifier training. On the QM9 molecular dataset, CANDI achieves competitive or superior frontiers compared to methods requiring specialized classifiers.

Figure 10: Frontiers for MDLM, UDLM, and CANDI with classifier-based guidance. CANDI achieves competitive results using generic classifiers.

Analysis of Frontier Behavior

A key methodological contribution is the use of frontier analysis (entropy-perplexity curves) rather than single-point metrics, as temperature tuning can dramatically alter relative rankings between models.

Figure 11: Temperature sweeps reveal optimal frontiers for each method; CANDI achieves its best performance at $\tau=0.7$.

Additionally, increasing NFE beyond 128 does not significantly expand the generative frontier for any method, indicating diminishing returns for additional steps.

Figure 12: Achievable frontiers for MDLM, DUO, and CANDI at small and large NFE. Beyond 128 NFE, further increases do not yield significant improvements.

Theoretical and Practical Implications

Theoretical Implications:

• The token identifiability framework provides a principled explanation for the failure of continuous diffusion in discrete domains with large vocabularies.
• The temporal dissonance phenomenon is a fundamental barrier that cannot be overcome by architectural tweaks alone; it requires explicit decoupling of discrete and continuous corruption.

Practical Implications:

• CANDI enables efficient, high-quality generation in discrete domains, particularly at low NFE, making it suitable for latency-sensitive applications.
• The hybrid approach allows for straightforward integration of classifier-based guidance, facilitating controllable generation without retraining classifiers.
• The approximate inference algorithm ensures scalability to large vocabularies and long sequences.

Future Directions

• Further exploration of optimal schedules for discrete and continuous corruption, beyond linear scaling.
• Extension of the hybrid framework to embedding diffusion and other discrete-continuous domains.
• Investigation of scaling behavior for larger models and more complex tasks, including code and multimodal generation.
• Integration with advanced sampling and distillation techniques to further reduce inference cost.

Conclusion

CANDI provides a theoretically grounded and empirically validated solution to the longstanding challenge of applying continuous diffusion models to discrete data. By decoupling discrete and continuous corruption mechanisms, it enables simultaneous learning of conditional structure and continuous score functions, unlocking the benefits of continuous geometry for discrete generative modeling. The framework is broadly applicable and sets a new standard for hybrid diffusion in discrete domains.