- The paper reveals that diffusion LLMs exhibit significantly higher hallucination rates and lower factual accuracy compared to AR-LLMs.
- The study employs controlled architectural and parametric alignments with standardized decoding and a modified HalluLens benchmark to ensure fair comparison.
- Practical implications stress the need for advanced error correction strategies to mitigate diffusion-specific failures such as context intrusion and incomplete denoising.
Uncovering Hallucination Patterns and Failure Modes in Diffusion LLMs
Introduction
This work provides a comprehensive and controlled comparative analysis of hallucination phenomena in Diffusion LLMs (dLLMs) versus their auto-regressive (AR) counterparts. By decoupling architectural, parametric, and decoding confounders, the study isolates the intrinsic impacts of the diffusion generation paradigm on reliability and output faithfulness. The investigation is motivated by recent parity in core benchmarks between dLLMs and AR-LLMs, prompting critical assessment of trustworthiness aspects that remain underexplored in diffusion-based models—specifically, the nature and prevalence of hallucinations.
Methodology
To attribute behavioral differences to generative mechanisms, two control group strategies are defined: (1) architectural alignment (e.g., LLaDA-8B vs. LLaMA-3-8B) and (2) parametric alignment (e.g., Dream-7B, initialized from Qwen2.5-7B). The comparison employs strictly pre-trained models, excluding instruction-tuning confounders. Decoding is standardized: AR-LLMs utilize greedy decoding, while dLLMs perform canonical diffusion with denoising steps equal to sequence length and temperature fixed to zero.
Hallucination is assessed using a variant of the HalluLens benchmark across three extrinsic tasks—Precise WikiQA, Long Wiki, and NonExistentRefusal—corresponding to factual recall, factual consistency in long-form generation, and boundary refusal for non-existent entities. The evaluation leverages an LLM-based automated judge (Gemini-2.5-Flash) validated with human annotation for robustness.
Main Results
dLLMs consistently exhibit higher hallucination rates and lower factual accuracy than their AR-LLM baselines across all tasks, even under tight controls for architecture and parametric knowledge. In the Precise WikiQA task, Dream-7B (dLLM) reports a Hallucination Rate of 92.54%, in contrast to 89.06% for Qwen2.5-7B (AR), and LLaDA-8B achieves only 3.92% Correct Rate versus 10.30% for LLaMA-3-8B. On long-form generation (Long Wiki), dLLMs deliver comparable recall but significantly lower precision, yielding inferior F1 scores. Crucially, for knowledge boundary detection, Dream-7B fails to refuse fabricated entities in 98.50% of cases.
These deficiencies underscore the persistence—and in some cases exacerbation—of generative errors in dLLMs, despite their theoretical advantages in global context access and bidirectional planning. Notably, the anticipated mitigation of AR-specific failure modes (e.g., snowballing effect, reversal curse) does not materialize. Instead, errors in early denoising are entrenched without an effective iterative correction mechanism, ultimately shifting rather than resolving the fundamental hallucination issue.
Inference Dynamics and Compute Trade-offs
dLLMs present divergent behaviors with respect to inference-time scaling. LLaDA-8B exhibits performance saturation, invariant under increased denoising steps, suggestive of a quasi-autoregressive operational paradigm induced by linear noise scheduling and high-confidence left-to-right decoding. Conversely, Dream-7B, leveraging minimum entropy decoding and non-sequential refinement, demonstrates monotonic improvement with added compute—reducing hallucination as denoising steps increase.
However, this capacity for enhancement is accompanied by emergent stability risks; non-sequential refinement increases the risk of unique error patterns such as context intrusion. These observations delineate a critical trade-off: quasi-AR dLLMs inherit AR rigidity and error accumulation, while unconstrained non-sequential models introduce new forms of instability.
Distinct Diffusion-Induced Failure Modes
Through detailed manual analysis, the study identifies failure modes specific to dLLMs:
- Premature Termination: Disjoint decoding segments, unable to resolve syntactic or semantic coherence, prompt early EOS or insertion of spurious boundaries.
- Incomplete Denoising: Particularly in rare or out-of-distribution scenarios, bidirectional attention struggles to reconcile incoherent anchors, resulting in nonsensical or placeholder-laden outputs.
- Context Intrusion: Bidirectional alignment on high-frequency or outlier tokens triggers abrupt topic shifts, hijacking the intended query path.
Quantitative annotation reveals incomplete denoising dominates in LLaDA-8B, while context intrusion is more prevalent in Dream-7B, matching their respective decode dynamics. The prevalence of these failure modes highlights the necessity for advanced sequence editing approaches—such as iterative re-masking or targeted insertion/deletion steps—currently absent from dLLM architectures.
Practical and Theoretical Implications
The findings have immediate impact on the deployment of dLLMs, given their augmented hallucination propensities and unique failure signatures. For practical applications requiring rigorous factuality and reliable refusal of out-of-distribution prompts—e.g., medical, legal, scientific domains—current dLLMs do not offer sufficient robustness. This necessitates renewed research focus on error correction within the diffusion paradigm, including inference-time methods (re-masking, self-reflective decoding) and architectural innovations enabling genuine global coherence enforcement.
Theoretically, the results delineate the diffusion paradigm’s intrinsic challenges: while theoretically equipotential with respect to sequence modeling, the absence of enforced monotonicity or robust error correction amplifies the persistence and idiosyncrasy of hallucinations, unless compensated by secondary mechanisms. Furthermore, the observed breakdown of proposed AR-specific error explanations (e.g., snowballing, reversal) suggests the need to revisit the fundamental taxonomy of generative failures in language modeling.
Future Directions
Key avenues for future research include:
- Integration of dynamic re-masking and self-reflective error correction during both training and inference.
- Systematic evaluation of instruction-tuned and alignment-tuned dLLMs, which current results show to be highly variable due to confounded safety and refusal strategies.
- Development of probabilistic control flows and decode strategies better leveraging the global planning potential of dLLMs without sacrificing structural coherence.
Conclusion
This controlled empirical investigation demonstrates that current dLLMs, despite achieving performance parity with AR-LLMs on core benchmarks, are inherently more susceptible to hallucination and exhibit characteristic failure patterns that undermine model reliability. Resolving these limitations is contingent on architectural and inference advances transcending naïve denoising, motivating significant methodological and theoretical work on robust, globally faithful non-autoregressive sequence generation.
For full details and reproducibility, the authors have made their evaluation code and protocols public: https://github.com/ZeroLoss-Lab/Lost-in-Diffusion.
