Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning

Abstract: Recent diffusion LLMs (dLLMs) have demonstrated both effectiveness and efficiency in reasoning via a block-based semi-autoregressive generation paradigm. Despite their progress, the fixed-size block generations remain a critical bottleneck for effective and coherent reasoning. 1. From a global perspective, different reasoning tasks would correspond to different optimal decoding block sizes, which makes a ``one-size-fits-all'' assumption ineffective. 2. Even within a single reasoning task, the rigid block partitioning would break the logical flow and reduce reasoning coherence. Through empirical observations, we reveal that for block-wise entropy, incorrect reasoning exhibits a fluctuating and unsteady trend between blocks, whereas the correctly generated tasks follow a consistent descending trend. Therefore, this paper proposes b1, a novel post-training framework for dLLMs that learns dynamic-size reasoning blocks via a Monotonic Entropy Descent objective with reinforcement learning to enhance reasoning coherence.b1 integrates seamlessly as a plug-and-play module with existing dLLM's post-training algorithms. Extensive experiments across various reasoning benchmarks showcase b1's consistent improvement over existing fixed-size block baselines. Our code has been released at https://github.com/YanJiangJerry/Block-R1.

• The paper introduces b1 that dynamically adjusts reasoning block sizes via reinforcement learning and a monotonic entropy descent objective to enhance reasoning coherence.
• It integrates an explicit indicator token to mark block boundaries, aligning dynamic blocks with semantic reasoning steps while delivering robust accuracy improvements.
• Experimental results on benchmarks, including a +19.53% improvement on Countdown, validate that b1 achieves coherent reasoning without significant computational overhead.

Dynamic-size Reasoning Blocks for Diffusion LLMs via Monotonic Entropy Descent with RL

Introduction and Motivation

Diffusion LLMs (dLLMs) have become a promising alternative to autoregressive (AR) LLMs by facilitating semi-autoregressive, block-wise parallel generation. Despite improvements in reasoning performance, the conventional use of fixed-size blocks in dLLMs introduces critical limitations in reasoning coherence and efficacy. The optimal block size is highly task-dependent, yet existing systems typically employ a universal block size, resulting in mismatches between block boundaries and logical reasoning steps. Empirical analysis reveals that correct reasoning traces exhibit monotonic entropy descent across blocks, while incorrect ones show irregular entropy patterns, indicating disrupted reasoning coherence.

Methodology

This paper presents b1, a novel post-training RL framework that learns to generate dynamic-size reasoning blocks aligned with semantic reasoning steps, thereby overcoming the rigid segmentation of fixed-size generation. The dynamic block definitions are learned using a Monotonic Entropy Descent (MED) objective—incorporated into RL training by a block entropy reward function that enforces monotonic entropy reduction across blocks. The approach is instantiated as a plug-and-play layer atop existing RL-driven dLLM training algorithms, including Diffu-GRPO, GDPO, d1, and wd1.

Dynamic Block Boundary Prediction

Central to b1 is an indicator token ($T_{\text{end}}$), denoting the end of a reasoning step, which the model learns to generate explicitly. Block boundaries are then dynamically assigned during generation by detecting this indicator, adapting block sizes to the underlying reasoning semantics. The indicator reward term ($R_{\text{ind}}$) encourages non-trivial, multi-step chains of thought, penalizing degenerate or trivially short reasoning traces.

Monotonic Entropy Descent Objective

Empirical results demonstrate a strong correlation between correct reasoning and monotonic entropy descent (block-level, token-averaged entropy). To formalize this, the MED reward is constructed as a relaxation of the negative Spearman’s rank correlation coefficient over the sequence of block entropies, converted into a dense, local pairwise comparison reward. The authors provide a rigorous proof that maximizing this local objective is equivalent, in expectation, to enforcing strict monotonicity globally.

The total RL objective for b1 thus combines:

• Local monotonic entropy descent reward ($R_{\text{ent}}$)
• Indicator block-ending reward ($R_{\text{ind}}$)
• Task-specific correctness/format rewards ($R_{\text{task}}$) All reward coefficients are set to one, demonstrating the plug-and-play property of b1 without custom hyperparameter optimization for each task.

Experimental Analysis

Experiments are conducted on LLaDA-8B-Instruct, using GSM8K, MATH500, Sudoku, and Countdown reasoning benchmarks. Comparisons are made to state-of-the-art RL fine-tuning algorithms (Diffu-GRPO, GDPO, d1, wd1) as well as AdaBlock-dLLM, a confidence-threshold inference-time truncation approach.

Main Results

Across all tasks and RL baselines, adding b1 yields consistent and often substantial improvements in reasoning accuracy, most notably on the challenging Countdown dataset, with up to +19.53% absolute improvement using wd1 + b1. On all benchmarks, the transformation to dynamic-size blocks yields superior performance relative to fixed-size or heuristic-truncated approaches. Zero-shot pass@1 accuracy demonstrates that b1 does not degrade computational efficiency, with minimal overhead in training and inference throughput.

Mechanistic Explanations

Ablation studies correlate the performance increment directly with the proposed rewards: omitting the entropy descent or the indicator reward produces a distinct loss of accuracy, particularly on high-reasoning-load tasks. Quantitative analysis shows that b1 increases the proportion of generations with positive rscc (i.e., monotonic block entropy descent), which correlates strongly with correct reasoning. Further, b1 is particularly effective at correcting high-entropy, “difficult” samples that fixed-size baselines consistently fail.

Case studies on representative mathematical and logic tasks demonstrate that fixed-size blocks fragment logical steps, introduce high entropy and repeated trivial token generation, and disrupt reasoning traces. In contrast, b1’s dynamic blocks align precisely with semantic steps, achieving coherent reasoning and correct results with monotonic entropy reduction.

Efficiency and Training Dynamics

Training and inference overheads for b1 are minimal because dynamic-block boundary detection introduces only a negligible computational cost compared to the self-attention operations that dominate dLLM complexity. Empirically, b1 achieves fast convergence, stable training dynamics, and high throughput.

Theoretical Implications

The work advances theoretical understanding of the connection between entropy dynamics and reasoning quality in diffusion-based text generation, formalizing monotonic entropy descent as a sufficient statistic of reasoning coherence. Theoretical contributions include proof that maximizing local block entropy drop rewards is equivalent to maximizing negative Spearman correlation globally, guaranteeing strict monotonicity under optimality.

Crucially, the approach leverages fine-grained, logit-level reward signals, enabling denser and more informative feedback than conventional sentence- or answer-level RL objectives. This aligns reward structure with the continuous uncertainty reduction paradigm inherent to reasoning in dLLMs.

Implications and Future Directions

The demonstration that dynamic block sizes aligned with semantic reasoning steps can be efficiently learned and enforced via RL and entropy-based rewards has several ramifications:

• Practical: The b1 framework is directly compatible with existing block-based dLLM architectures and state-of-the-art RL fine-tuning strategies, requiring no architectural changes and delivering immediate, robust improvements in reasoning performance, especially for domains requiring fine-grained, multi-step reasoning.
• Theoretical: It establishes monotonic entropy descent as a necessary condition for block-wise reasoning coherence, offering a new lens for analysis of generative uncertainty in LLMs. The explicit decoupling of block boundaries from rigid token counts paves the way for investigations into more generalized, adaptive generative schemes in diffusion models.
• Further research: Extensions include integrating b1 with multimodal dLLMs, exploring finer block-boundary control (e.g., by predicting soft or hierarchical boundaries), and applying the MED principle to other parallel-generation paradigms or reasoning-intensive domains such as program synthesis, formal proof systems, and complex multi-hop question answering.

Conclusion

The b1 framework conclusively addresses the mismatch between rigid block segmentation and semantic reasoning process in diffusion LLMs. By learning dynamic-size blocks through monotonic entropy descent objectives in reinforcement learning, it enables generation aligned with logical reasoning steps, producing coherent, accurate reasoning traces. The gains are robust, efficient, and compatible with existing RL and dLLM methods, setting a foundation for further advancements in block-based massively parallel LLM generation and training.