DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference

Abstract: Diffusion LLMs (DLMs) have emerged as a promising alternative to autoregressive language generation due to their potential for parallel decoding and global refinement of the entire sequence. To unlock this potential, DLM inference must carefully balance generation quality and decoding speed. Recent block-wise DLM decoding methods improve this trade-off by performing diffusion-based decoding sequentially in blocks. However, existing methods typically rely on fixed block schedules or current-step local signals to determine block boundaries, and use conservative confidence-based parallel decoding to avoid conflicts, limiting the quality-speed trade-off. In this paper, we argue that block-wise DLM inference requires more suitable signals for its two core decisions: cross-step signals for determining block boundaries, and token-level conflict signals for parallel decoding. Based on this view, we propose DepCap, a training-free framework for efficient block-wise DLM inference. Specifically, DepCap instantiates the cross-step signal as the influence of the last decoded block and uses it to adaptively determine how far the next block should extend, while identifying a conflict-free subset of tokens for safe parallel decoding within each block, enabling substantial inference acceleration with negligible quality degradation. DepCap is a plug-and-play method applicable to various DLMs, and compatible with existing KV-cache strategies for block-wise DLM. An information-theoretic analysis further suggests that the cumulative last-block influence on a candidate block is approximately additive across tokens, supporting the proposed block-partitioning criterion. Experimental results show that DepCap achieves favorable speed-quality trade-offs across multiple DLM backbones and reasoning and coding benchmarks, with up to 5.63$\times$ speedup without significant performance degradation.

• The paper introduces DepCap, a training-free framework that leverages cross-step dependency signals to dynamically determine block boundaries.
• It employs token-level conflict detection for aggressive parallel decoding, achieving up to 5.63× speedup and a 7.4% accuracy boost on benchmarks.
• Empirical results show that DepCap robustly balances speed and quality across diverse diffusion LM architectures and integrates smoothly with cache strategies.

DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LLM Inference

Introduction

Diffusion LLMs (DLMs) have recently emerged as a compelling alternative to traditional autoregressive approaches for sequence generation, particularly due to their potential for parallel token refinement and global sequence modeling. The inherent challenge of DLM inference lies in the trade-off between generation quality and inference efficiency, as the multi-step denoising process and global refinement requirements can substantially increase computational latency. Block-wise decoding schemes address this problem by partitioning the sequence into manageable blocks, allowing for more scalable parallel decoding and compatibility with KV-caching strategies. However, existing block-wise solutions are fundamentally limited by their reliance on fixed block boundaries or heuristic, local, per-step signals, and often apply conservative, confidence-only parallel decoding that fails to optimally exploit sequence structure or token dependencies, thus impeding the attainable balance between speed and generation quality.

DepCap Framework

DepCap introduces a training-free, dependency-aware block-wise decoding paradigm for DLMs, leveraging cross-step dependency signals for adaptive block formation and token-level conflict detection for safe, aggressive parallel decoding.

The high-level approach consists of two principal modules:

1. Dependency-Guided Adaptive Block Partitioning (DepGA-Block): Block boundaries are dynamically determined by measuring the cross-step influence of the previous decoded block using the Kullback-Leibler divergence between predictive token distributions before and after block updates, supplemented by predictive uncertainty (Shannon entropy). The composite dependency score enables block boundaries to be selected where model uncertainty outweighs the contextual support from the preceding block. This mechanism adapts block size in response to the denoising process, instead of relying on static schedules or only local signals.
2. Conflict-Aware Parallel Decoding (CAP-Decoding): Within each adaptively selected block, CAP-Decoding identifies a conflict-free subset of masked tokens that can be decoded in parallel while minimizing the risk of token dependencies introducing generation errors. Conflicts between token predictions are quantified via a pairwise log-probability-based score, filtering out candidates whose states are highly entangled. This enables aggressive parallelism without substantial quality degradation, moving beyond naive confidence-based strategies.

   Figure 1: Comparison of traditional block-wise DLM inference using fixed schedules versus DepCap's adaptive block formation and conflict-aware parallel token decoding. DepGA-Block utilizes last-block influence and predictive uncertainty to decide boundaries, CAP-Decoding selects conflict-free token subsets for parallel updates.

Notably, DepCap is training-free and plug-and-play—it can be flexibly applied to various DLM backbones and works seamlessly with established block-wise cache strategies (e.g., Prefix Cache, Dual Cache).

Theoretical Underpinnings

DepGA-Block's dependency score is theoretically justified via an information-theoretic analysis. The cross-step influence is linked to the conditional mutual information between the last decoded block and prospective block positions, supporting the additivity assumption under limited local overlap. The block expansion criterion is therefore motivated as a balance between cumulative support (cross-step influence) and cumulative uncertainty (predictive entropy), with expansion truncated when the former is no longer dominant. This provides both a principled probabilistic interpretation and a tractable implementation pathway.

Empirical Results

Experiments span multiple contemporary diffusion LM architectures (e.g., LLaDA-8B-Instruct, Dream-v0-base-7B, LLaDA-1.5) and evaluation on diverse benchmarks targeting mathematical reasoning (GSM8K, Math-500) and code generation (MBPP, HumanEval).

Key empirical findings include:

• Accelerated Inference: DepCap delivers up to 5.63$\times$ speedups over baseline decoding on MBPP using LLaDA-1.5, while increasing accuracy by 7.4% absolute—demonstrating that substantial throughput gains can be achieved with even improved output quality.
• Stable Quality-Sensitivity Trade-off: Averaged over eight task-model combinations, DepCap achieves a 3.57$\times$ speedup relative to vanilla LLaDA decoding with negligible accuracy loss.
• Cache Compatibility: The method remains effective when integrated with existing cache mechanisms (Prefix/Dual Cache), achieving the best throughput-accuracy trade-off among highly competitive baselines.
• Module Contribution: Ablation studies confirm that both cross-step adaptive block selection and conflict-aware parallel decoding are independently beneficial, and their combination maximizes speed-up without compromising accuracy.
• Robustness: Hyperparameter sweeps across generation length, maximum block size, uncertainty weighting, and conflict/threshold cutoffs show DepCap's effectiveness is robust and not reliant on narrow parameter choices.

Implications and Future Directions

DepCap's approach—adapting block granularity via cross-step dependency analysis and resolving intra-block token conflicts—enables DLMs to approach the practical efficiency of autoregressive models while retaining the theoretical advantages of global refinement. The techniques are general and can, in principle, be adapted to both next-generation DLM architectures and alternative parallel sequence generation paradigms.

Of particular theoretical interest, the information-theoretic block selection criterion opens new research into principled signal selection for block formation beyond text, e.g., multimodal or structured outputs. Practically, the successful demonstration of large speed-ups at scale with strong accuracy bodes well for the deployment of DLMs in high-latency, high-throughput environments, such as code synthesis, interactive agents, or real-time reasoning.

Future work will likely extend these cross-step adaptive partitioning strategies further, e.g., to sliding-window or dynamic-block paradigms, and seek tighter probabilistic bounds on dependency score behavior. Bridging to more expressive conflict and dependency modeling is another salient direction, particularly using structured or learned dependency graphs.

Conclusion

DepCap presents a principled and training-free framework for block-wise DLM inference, introducing cross-step dependency-guided block formation and conflict-aware parallel decoding. Supported by information-theoretic analysis and extensive empirical validation, the approach achieves strong throughput gains with no substantive compromise on generation quality. DepCap establishes new state-of-the-art for efficient DLM inference and provides foundational ideas for future adaptive parallel decoding research.

Figure 1: The DepCap framework’s adaptive block boundary selection and conflict-aware token decoding significantly improve the DLM inference speed-quality trade-off compared to fixed block schedules.