---
title: 'DreamReasoner-8B: Block Diffusion Reasoning'
url: https://www.emergentmind.com/topics/dreamreasoner-8b
type: topic
---

# DreamReasoner-8B: Block Diffusion Reasoning

Searching arXiv for the primary DreamReasoner-8B paper and closely related Dreamer work.
DreamReasoner-8B is an 8-billion-parameter, transformer-based diffusion language model for long chain-of-thought reasoning that implements block diffusion with parallel, block-wise denoising [2606.19257]. It is initialized from Qwen3-8B-Base and continually pretrained and fine-tuned on math and code tasks while preserving autoregression across blocks and performing local bidirectional denoising within each block [2606.19257]. In this formulation, blocks are contiguous spans of tokens; earlier blocks are treated as clean, causal context, and the current block is denoised in parallel under a discrete, absorbing corruption process with a dedicated [MASK] token [2606.19257]. The model’s central methodological claim is that training granularity strongly affects long-CoT reasoning: direct training with large block sizes degrades reasoning, whereas small-block training preserves it, and block-size curriculum learning reconciles efficiency with reasoning fidelity across diverse inference block sizes [2606.19257].

## 1. Conceptual definition and modeling objective

DreamReasoner-8B is a block diffusion language model rather than a conventional left-to-right autoregressive model. Its generative factorization preserves autoregression across blocks while replacing token-by-token generation inside each block with iterative parallel denoising [2606.19257]. The sequence is partitioned into $K$ blocks, each of length $B$, and the joint distribution is written as
$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$
[2606.19257].

Within block $b^k$, the forward process applies an absorbing [MASK] corruption only to the current block while keeping the prefix context $b^{<k}_0$ clean [2606.19257]. Training is discrete token diffusion with cross-entropy on masked positions rather than continuous embedding diffusion or score matching. The loss is
$$
\begin{equation}
\begin{split}
\mathcal{L}(\theta) \!=\!
-\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t
\log p_{\theta}\bigl(b^{k}_0 \!\mid\! b^{<k}_0\!,b^k_t\bigr)\biggr]\!, \end{split}
\label{eq:block-loss}
\end{equation}
$$
with $t \sim \operatorname{Uniform}(0,1)$, $\alpha_t \in (0,1)$, $\alpha'_t = d\alpha_t/dt$, and $w_t = \alpha'_t/(1-\alpha_t)$ [2606.19257].

This design localizes denoising to blocks while retaining causal scaffolding across blocks. A plausible implication is that DreamReasoner-8B occupies an intermediate point between purely autoregressive decoding and full-sequence diffusion: it preserves left-to-right global structure but relaxes local generation order to obtain intra-block parallelism.

## 2. Architecture, implementation, and training stack

DreamReasoner-8B has approximately 8B parameters and uses a transformer backbone initialized from Qwen3-8B-Base [2606.19257]. Training and inference are implemented in Megatron-LM, extended to support block diffusion with FlexAttention kernels for structured sparse attention, while SGLang is used as the inference engine [2606.19257]. The model uses standard transformer tooling and maximum generation length in experiments is 24K tokens, with 8,192 tokens in the pilot study [2606.19257].

Its operational unit is the token block. During generation of block $k$, the prefix $b^{<k}_0$ is fixed and clean, and the model starts from a fully masked $b^k_t$ [2606.19257]. It then iteratively predicts masked positions conditioned on the prefix and refines them in parallel until the block is fully unmasked [2606.19257]. This yields high intra-block parallelism while maintaining inter-block causal coherence [2606.19257].

Continual pretraining uses the block-diffusion objective with block size 32 on a high-quality corpus of 160B tokens curated from open sources such as OLMo3 and Nemotron Nano V3 [2606.19257]. Loss is computed only on masked positions, and a noise injection module enforces at least one masked token per block to prevent degenerate unmasked cases [2606.19257]. Supervised fine-tuning uses PromptCoT 2.0 with 4.8M samples and context length 16,384 [2606.19257]. The paper does not report GPUs, FLOPs, batch sizes, or exact diffusion step counts [2606.19257].

The DreamReasoner-8B naming convention is distinct from the earlier Dreamer framework of depth-recurrent attention mixtures, despite the similarity in nomenclature. The earlier Dreamer work concerns depth-recurrent latent reasoning with sequence attention, depth attention, and sparse expert attention [2601.21582]. This suggests that “DreamReasoner” in the 8B system should be understood specifically as a block diffusion reasoning model rather than as an 8B instantiation of the Dreamer depth-recurrent architecture.

## 3. Block-size curriculum learning

The defining methodological contribution of DreamReasoner-8B is block-size curriculum learning [2606.19257]. The paper’s analysis reports a stark performance disparity: training with large block sizes yields remarkably poor reasoning, whereas small block sizes preserve effective reasoning [2606.19257]. The proposed remedy is to begin with fine-grained block sizes and gradually transition to coarser ones, so that the model first learns local causal structure and then acquires robustness under more parallel generation [2606.19257].

Two schedules are reported. In the pilot study, training lasts 4 epochs total: 3 epochs at block size 4, then 1 epoch at block size 32 [2606.19257]. In final DreamReasoner-8B training on PromptCoT 2.0, the model is trained for 3 epochs at block size 4 and then enters a mixed-granularity stage where the block size for each sample is randomly drawn from $\{4, 8, 16, 32\}$ [2606.19257]. The mixed stage intentionally preserves small blocks to avoid degradation during small-block inference [2606.19257]. Transition criteria are fixed by epochs; no validation-driven or adaptive schedule is used [2606.19257].

The paper characterizes a “flexibility trap” in diffusion decoding: arbitrary denoising orders can bypass high-uncertainty tokens, and strict autoregressive decoding partially recovers performance for models trained only with large blocks [2606.19257]. This is presented as evidence that coarse blocks pressure the model to aggregate in parallel at the expense of token-level sequential fidelity [2606.19257]. A plausible implication is that the curriculum functions less as a generic optimization heuristic and more as a constraint-preserving mechanism that maintains local causal priors while broadening the admissible inference granularity.

## 4. Inference procedures and decoding regimes

At deployment, inference block size is chosen to trade off efficiency and accuracy, and the model generalizes across $B \in \{4,8,16,32\}$ [2606.19257]. The principal diffusion decoding method is LowConfidence decoding. For the current block, tokens whose confidence exceeds a threshold $\tau = 0.95$ are committed, while the rest remain masked and are reconsidered in subsequent denoising passes [2606.19257]. Temperature is 0 for diffusion baselines under thresholded decoding [2606.19257].

The per-block diffusion procedure is specified as follows [2606.19257]:

1. Initialize $b^k$ as fully masked and keep $b^{<k}_0$ fixed.
2. Run a forward pass to obtain token-wise confidences within $b^k$.
3. Commit tokens meeting the criterion: $c_i \ge \tau$ for LowConfidence, or $c_i \ge \tau_i$ for RelaxedConfidence.
4. Repeat until all tokens in $b^k$ are committed, then move to block $k+1$.

The paper also evaluates a strict autoregressive baseline in which only the leftmost masked token is committed per step [2606.19257]. This is not the main intended operating mode, but it serves as an analytical control for measuring the effect of local causal ordering.

A further decoding variant, RelaxedConfidence, is described as an analytical probe to accelerate block-wise decoding [2606.19257]. For masked position $i$, with neighborhood $\mathcal{N}_r(i)=\{j:0<|i-j|\le r\}$ and reliable set $\mathcal{R}=\mathcal{D}\cup\{j:c_j\ge\kappa\}$, the position-specific support and threshold are
\[
S_i =
\frac{
\sum_{j\in\mathcal{N}_r(i)\cap\mathcal{R}} w_{i,j}
}{
\sum_{j\in\mathcal{N}_r(i)} w_{i,j}
}, \quad
\tau_i = \tau - (\tau-\tau_{\min})S_i .
\]
The reported settings are $\tau=0.95$, $r=4$, $\kappa=0.7$, and $\tau_{\min}=0.7$ [2606.19257]. Tokens with stronger local support can therefore be committed earlier at a lower effective threshold [2606.19257].

## 5. Empirical performance

DreamReasoner-8B is evaluated on mathematical and code reasoning benchmarks and is reported as competitive with leading open autoregressive models such as Qwen3-8B [2606.19257]. For the base model, the paper reports the following scores: MMLU 75.4, ARC-E 87.0, ARC-C 63.5, HellaSwag 76.6, PIQA 81.2, WinoGrande 74.2, RACE 43.8, GSM8K 83.4, MATH 55.8, GPQA 42.9, HumanEval 69.5, and MBPP 71.7 [2606.19257]. DreamReasoner-8B-Base outperforms full-sequence diffusion baselines such as Dream-v0-7B-Base and LLaDA-8B-Base and is competitive with autoregressive base models such as Qwen3-8B-Base [2606.19257].

The continual pretraining ablation from Qwen3-8B-Base isolates the value of block diffusion. The reported scores are shown below.

| Model | GSM8K | MATH | HumanEval | MBPP |
|---|---:|---:|---:|---:|
| Qwen3-8B-Base | 86.5 | 52.7 | 65.8 | 68.8 |
| DreamReasoner-8B-Base | 83.5 | 51.2 | 62.8 | 60.4 |
| w/o block diffusion | 75.5 | 40.6 | 31.1 | 41.6 |

These results indicate that block diffusion substantially outperforms full-sequence diffusion for reasoning-intensive tasks [2606.19257].

The pilot study directly measures the interaction of training block size, inference block size, and decoding strategy. Train-$B=4$ models remain comparatively robust across inference granularities, while train-$B=32$ models degrade sharply under diffusion decoding on complex tasks [2606.19257]. For example, Train $B=32 \rightarrow$ Infer $B=32$ yields AIME25 21.3 under LowConfidence, whereas Curriculum $4 \rightarrow 32 \rightarrow$ Infer $B=32$ yields 38.3 [2606.19257]. Likewise, Curriculum $4 \rightarrow 32 \rightarrow$ Infer $B=4$ gives AIME25 43.8 under LowConfidence [2606.19257].

For the final DreamReasoner-8B under PromptCoT 2.0 SFT with LowConfidence $\tau=0.95$, the reported results across inference block sizes are:

| Inference block size | AIME24 | AIME25 | LiveCodeBench |
|---|---:|---:|---:|
| $B=4$ | 73.8 | 65.0 | 51.3 |
| $B=8$ | 71.7 | 64.6 | 53.9 |
| $B=16$ | 71.7 | 62.9 | 53.6 |
| $B=32$ | 68.3 | 63.3 | 50.4 |

The paper compares these to Qwen3-8B-Thinking at 76.0, 67.3, and 52.8 on AIME24, AIME25, and LiveCodeBench respectively [2606.19257]. It also reports stronger performance than several other open 7B–8B autoregressive baselines, including DeepSeek-R1-Distill-Qwen-7B, MiMo-7B-RL, and AceReason-Nemotron-1.1-7B [2606.19257]. Relative to diffusion baselines, DreamReasoner-8B is reported to generalize across inference block sizes while SDAR-30B-A3B-Sci collapses as block size grows [2606.19257].

## 6. Efficiency characteristics

The efficiency analysis centers on tokens-per-forward-pass (TPF), defined as generated tokens divided by total forward passes, including token-prediction passes and any KV-cache recomputation [2606.19257]. Relative to autoregression, block-wise parallel denoising increases TPF because multiple positions can be committed within one denoising pass [2606.19257].

The paper reports that TPF increases monotonically with larger inference block sizes while accuracy remains stable for DreamReasoner-8B [2606.19257]. For $B=32$ and LowConfidence $\tau=0.95$, TPF rises across the response from about 1.2–1.6 in early blocks to about 2.8–3.5 in later blocks as more context accumulates [2606.19257]. Threshold sensitivity is nontrivial: lower $\tau$ increases TPF but risks accepting low-confidence tokens, and code is particularly sensitive to aggressive thresholds [2606.19257].

RelaxedConfidence improves throughput without material accuracy loss according to the reported measurements [2606.19257]. During thinking, AIME 2025 TPF increases from about 2.1 to about 2.8, and LiveCodeBench from about 1.7 to about 1.9 [2606.19257]. During answering, AIME 2025 rises from about 2.4 to about 3.6, and LiveCodeBench from about 3.9 to about 6.2 [2606.19257]. Earlier summary statistics in the paper describe average TPF gains of 22.5% and 54.5% for thinking and answering phases under relaxed thresholds [2606.19257].

The paper does not report wall-clock times [2606.19257]. Accordingly, its efficiency claims are framed in terms of decoding granularity and TPF rather than end-to-end latency. This suggests that DreamReasoner-8B’s practical advantage depends not only on model quality but also on implementation details in Megatron-LM, FlexAttention, and SGLang.

## 7. Interpretation, limitations, and relation to broader research

The qualitative analysis attributes the failure of large-block-only training to overfitting toward intra-block parallel aggregation, which compromises token-level sequential reasoning and makes diffusion decoding brittle on complex tasks [2606.19257]. The fact that strict autoregressive decoding partially recovers performance in those models is used to argue that local causal dependencies remain critical for long-form reasoning even in a diffusion setting [2606.19257]. The curriculum is therefore presented as a way to preserve fine-grained local causality while adding robustness to coarse-grained, parallel generation [2606.19257].

Several limitations are explicit. Content-adaptive block partitioning is not explored, though the paper identifies it as a possible route to improving the efficiency-quality frontier [2606.19257]. Evaluation is concentrated on math and code rather than broader domains such as tool use and agentic coding [2606.19257]. Decoding remains bottlenecked by conservative per-token thresholds, and the paper identifies principled accept/reject mechanisms and stability analyses for larger models and longer contexts as open questions [2606.19257].

Recommended usage settings are also domain-specific. For math CoT tasks, $B \in \{8,16\}$ is reported to balance accuracy and parallelism, with $\tau=0.95$ for conservative decoding; for latency-critical settings, $B=32$ with $\tau=0.95$ is suggested, with only modest accuracy drop for DreamReasoner-8B [2606.19257]. For code generation, $B \in \{8,16\}$ with $\tau=0.95$ is recommended, and aggressive threshold reductions are discouraged unless RelaxedConfidence is used [2606.19257].

In the landscape of reasoning models, DreamReasoner-8B is significant because it provides a concrete, open-source recipe for reasoning-capable diffusion language modeling at the 8B scale [2606.19257]. Its main technical thesis is not merely that diffusion can be applied to reasoning, but that the training granularity of denoising must be aligned with the causal structure required by long-CoT generation. A plausible implication is that future diffusion reasoners may be differentiated less by raw parameter scale than by the degree to which their training curriculum, block structure, and commit criteria preserve sequential fidelity under parallel decoding.

Source: https://www.emergentmind.com/topics/dreamreasoner-8b