Residual Context Diffusion Language Models

Abstract: Diffusion LLMs (dLLMs) have emerged as a promising alternative to purely autoregressive LLMs because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a "remasking" mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ~1 billion tokens. RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4-5x fewer denoising steps at equivalent accuracy levels.

• The paper introduces an entropy-weighted residual mechanism that recycles discarded token signals to enhance parallel decoding efficiency.
• It demonstrates that RCD significantly reduces sequential iterations and nearly doubles Pass@1 accuracy on benchmarks like AIME25.
• Empirical results show robust scalability and stable training, positioning RCD as an effective alternative to autoregressive and latent reasoning methods.

Residual Context Diffusion LLMs: Mechanism, Empirical Analysis, and Theoretical Implications

Introduction and Motivation

Diffusion LLMs (dLLMs) provide a computationally attractive alternative to autoregressive (AR) decoding, enabling parallel token generation via iterative masked denoising. However, current dLLMs rely on a remasking procedure: only the most confident tokens are committed at each step, with remaining tokens reset and their computed statistics discarded. This approach yields significant inefficiencies, as discarded token predictions contain nontrivial semantic signals pertinent to subsequent decoding iterations. The present work introduces Residual Context Diffusion (RCD), a mechanism that recycles intermediate residuals, dynamically aggregates them into context vectors based on entropy, and injects them into subsequent denoising steps. This innovation leverages otherwise wasted computation, improving both decoding accuracy and efficiency, and narrowing the gap between AR and diffusion-based paradigms.

Figure 1: Overview of the residual denoising mechanism in Residual Context Diffusion (RCD); residuals are extracted from discarded tokens and injected into subsequent denoising steps.

Residual Context Diffusion: Methodology

RCD centers on the entropy-weighted residual mechanism. At each denoising iteration, token probabilities for undecoded (low-confidence) positions are aggregated into continuous embedding vectors via a weighted sum over the model’s vocabulary embeddings. The key innovation is the adaptive interpolation between original masked embeddings and residual vectors, governed by normalized Shannon entropy. High-entropy tokens inherently possess richer contextual information and, therefore, are weighted more heavily.

The model operates in two stages: a frozen Reference Model generates stable residual distributions; a Target Model utilizes these as context via standard cross-entropy supervision, avoiding computationally infeasible recursive backpropagation. During inference, RCD uses temperature-scales on entropy to align test-time dynamics with training, with an initial warm-start from the Reference Model followed by fully recursive decoding within the Target Model.

Figure 2: RCD inference and training schematic; residual distributions are gathered from uncommitted tokens and transmitted for recursive refinement, with a decoupled reference/target architecture stabilizing training.

Empirical Results

RCD models were fine-tuned and evaluated on a suite of mathematical reasoning tasks (GSM8K, MATH500, MinervaMath, and competition-level AIME24/25), encompassing both global-context bidirectional dLLMs (LLaDA) and block-wise semi-AR models (SDAR). Across all configurations, RCD consistently achieves superior accuracy compared to Sequential Denoising (SeqD) baselines, especially at higher block sizes and model scales. For instance, on the AIME25 benchmark, SDAR-8B-b64 with RCD yields a 19.79% Pass@1 accuracy, more than doubling the SeqD score (9.79%).

Figure 3: Intermediate recall analysis illustrates high recall@5 for early denoising steps, confirming that intermediate token distributions are semantically informative.

Furthermore, RCD is shown to recover substantial parallelism gains via the Pareto frontier of "accuracy versus tokens per denoising step": for a fixed accuracy, RCD models can generate several tokens per step, reducing the number of sequential iterations by 4–5× compared to SeqD at one-token-per-step. Through integration with production inference frameworks (Fastdllm, D2F), RCD maintains comparable throughput (tokens per second) to practical baselines while delivering mitigating accuracy improvements.

Figure 4: Accuracy-token Pareto analysis; RCD achieves substantial speedups, enabling high-accuracy decoding with increased parallelism per step.

Ablation and Trade-Offs

An ablation on the residual interpolation parameter $\alpha$ demonstrates that normalized entropy-based weighting achieves the fastest parallel decoding rate while preserving competitive accuracy. Alternative strategies (linear, max-probability, inverse mapping) fail to reach the optimal trade-off, either collapsing accuracy or restricting parallelism due to suboptimal exploitation of predictive uncertainty. The empirical evidence supports entropy as the appropriate transmission signal for residual context in sequence denoising.

Figure 5: Ablation of interpolation strategies ($\alpha$); entropy-based residual aggregation dominates in balancing accuracy and decoding efficiency.

Comparison to Latent Reasoning Methods and Scalability

Compared to methods such as Loopholing "Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall" (Jo et al., 22 Oct 2025), which propagate raw hidden state information, RCD offers robust scalability to large models and high block sizes. By grounding residual transmission in the model’s input embeddings rather than hidden layers, RCD avoids divergent semantic spaces and norm explosion, achieving stable training and inference. Data efficiency is a notable attribute: RCD reaches near-optimal performance within a single epoch of ∼300M tokens, outperforming existing latent-based methods that often fail under identical conditions.

Theoretical and Practical Implications

From a theoretical standpoint, RCD clarifies how information loss due to aggressive remasking in masked diffusion decoding can be mitigated by recycling discarded context. This relaxes the strict left-to-right dependency modeling problem and instead enables information propagation via residuals, improving semantic coherence and convergence in parallel decoding regimes. Practically, RCD's efficiency and accuracy enhancements enable scalable deployment of dLLMs in both latency-constrained and high-throughput settings, and its compatibility with inference frameworks positions it for widespread adoption. The entropy-weighted mechanism may also generalize to other latent reasoning and multimodal fusion domains where structured transmission of uncertainty is critical.

Future Directions

Potential future research includes applying RCD-style residual context transmission to other modalities (e.g., latent visual reasoning) and structurally adapting the temperature scaling and interpolation mechanisms for continual learning and lifelong reasoning. Further analysis of the theoretical bounds of residual context injection—especially in block-wise vs. bidirectional architectures—may elucidate the limits and optimization pathways for massively parallel generation.

Conclusion

Residual Context Diffusion constitutes a principled solution to the information bottleneck in masked diffusion LLMs, leveraging entropy-driven recycling of discarded signals to realize state-of-the-art accuracy, parallel decoding efficiency, and robust scalability. Its methodological contributions and empirical results position RCD as a predominant paradigm for high-fidelity, resource-effective language generation in future dLLM architectures.