DreamReasoner-8B: Block Diffusion Reasoning

• DreamReasoner-8B is a block diffusion language model that partitions token sequences into blocks and uses iterative, parallel intra-block denoising.
• It employs curriculum block-size training to preserve fine-grained causal structure while enabling efficient, global autoregressive context for varied inference granularities.
• Initialized from Qwen3-8B-Base and fine-tuned on math and code tasks, the model achieves competitive benchmark performance with improved tokens-per-forward-pass efficiency.

Searching arXiv for the primary DreamReasoner-8B paper and closely related Dreamer work. DreamReasoner-8B is an 8-billion-parameter, transformer-based diffusion LLM for long chain-of-thought reasoning that implements block diffusion with parallel, block-wise denoising (Wu et al., 17 Jun 2026). It is initialized from Qwen3-8B-Base and continually pretrained and fine-tuned on math and code tasks while preserving autoregression across blocks and performing local bidirectional denoising within each block (Wu et al., 17 Jun 2026). In this formulation, blocks are contiguous spans of tokens; earlier blocks are treated as clean, causal context, and the current block is denoised in parallel under a discrete, absorbing corruption process with a dedicated [MASK] token (Wu et al., 17 Jun 2026). The model’s central methodological claim is that training granularity strongly affects long-CoT reasoning: direct training with large block sizes degrades reasoning, whereas small-block training preserves it, and block-size curriculum learning reconciles efficiency with reasoning fidelity across diverse inference block sizes (Wu et al., 17 Jun 2026).

1. Conceptual definition and modeling objective

DreamReasoner-8B is a block diffusion LLM rather than a conventional left-to-right autoregressive model. Its generative factorization preserves autoregression across blocks while replacing token-by-token generation inside each block with iterative parallel denoising (Wu et al., 17 Jun 2026). The sequence is partitioned into $K$ blocks, each of length $B$, and the joint distribution is written as

$$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$

(Wu et al., 17 Jun 2026).

Within block $b^k$, the forward process applies an absorbing [MASK] corruption only to the current block while keeping the prefix context $b^{<k}_0$ clean (Wu et al., 17 Jun 2026). Training is discrete token diffusion with cross-entropy on masked positions rather than continuous embedding diffusion or score matching. The loss is

$\begin{equation} \begin{split} \mathcal{L}(\theta) \!=\! -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \!\mid\! b^{<k}_0\!,b^k_t\bigr)\biggr]\!, \end{split} \label{eq:block-loss} \end{equation}$

with $t \sim \operatorname{Uniform}(0,1)$, $\alpha_t \in (0,1)$, $\alpha'_t = d\alpha_t/dt$, and $w_t = \alpha'_t/(1-\alpha_t)$ (Wu et al., 17 Jun 2026).

This design localizes denoising to blocks while retaining causal scaffolding across blocks. A plausible implication is that DreamReasoner-8B occupies an intermediate point between purely autoregressive decoding and full-sequence diffusion: it preserves left-to-right global structure but relaxes local generation order to obtain intra-block parallelism.

2. Architecture, implementation, and training stack

DreamReasoner-8B has approximately 8B parameters and uses a transformer backbone initialized from Qwen3-8B-Base (Wu et al., 17 Jun 2026). Training and inference are implemented in Megatron-LM, extended to support block diffusion with FlexAttention kernels for structured sparse attention, while SGLang is used as the inference engine (Wu et al., 17 Jun 2026). The model uses standard transformer tooling and maximum generation length in experiments is 24K tokens, with 8,192 tokens in the pilot study (Wu et al., 17 Jun 2026).

Its operational unit is the token block. During generation of block $B$0, the prefix $B$1 is fixed and clean, and the model starts from a fully masked $B$2 (Wu et al., 17 Jun 2026). It then iteratively predicts masked positions conditioned on the prefix and refines them in parallel until the block is fully unmasked (Wu et al., 17 Jun 2026). This yields high intra-block parallelism while maintaining inter-block causal coherence (Wu et al., 17 Jun 2026).

Continual pretraining uses the block-diffusion objective with block size 32 on a high-quality corpus of 160B tokens curated from open sources such as OLMo3 and Nemotron Nano V3 (Wu et al., 17 Jun 2026). Loss is computed only on masked positions, and a noise injection module enforces at least one masked token per block to prevent degenerate unmasked cases (Wu et al., 17 Jun 2026). Supervised fine-tuning uses PromptCoT 2.0 with 4.8M samples and context length 16,384 (Wu et al., 17 Jun 2026). The paper does not report GPUs, FLOPs, batch sizes, or exact diffusion step counts (Wu et al., 17 Jun 2026).

The DreamReasoner-8B naming convention is distinct from the earlier Dreamer framework of depth-recurrent attention mixtures, despite the similarity in nomenclature. The earlier Dreamer work concerns depth-recurrent latent reasoning with sequence attention, depth attention, and sparse expert attention (Knupp et al., 29 Jan 2026). This suggests that “DreamReasoner” in the 8B system should be understood specifically as a block diffusion reasoning model rather than as an 8B instantiation of the Dreamer depth-recurrent architecture.

3. Block-size curriculum learning

The defining methodological contribution of DreamReasoner-8B is block-size curriculum learning (Wu et al., 17 Jun 2026). The paper’s analysis reports a stark performance disparity: training with large block sizes yields remarkably poor reasoning, whereas small block sizes preserve effective reasoning (Wu et al., 17 Jun 2026). The proposed remedy is to begin with fine-grained block sizes and gradually transition to coarser ones, so that the model first learns local causal structure and then acquires robustness under more parallel generation (Wu et al., 17 Jun 2026).

Two schedules are reported. In the pilot study, training lasts 4 epochs total: 3 epochs at block size 4, then 1 epoch at block size 32 (Wu et al., 17 Jun 2026). In final DreamReasoner-8B training on PromptCoT 2.0, the model is trained for 3 epochs at block size 4 and then enters a mixed-granularity stage where the block size for each sample is randomly drawn from $B$3 (Wu et al., 17 Jun 2026). The mixed stage intentionally preserves small blocks to avoid degradation during small-block inference (Wu et al., 17 Jun 2026). Transition criteria are fixed by epochs; no validation-driven or adaptive schedule is used (Wu et al., 17 Jun 2026).

The paper characterizes a “flexibility trap” in diffusion decoding: arbitrary denoising orders can bypass high-uncertainty tokens, and strict autoregressive decoding partially recovers performance for models trained only with large blocks (Wu et al., 17 Jun 2026). This is presented as evidence that coarse blocks pressure the model to aggregate in parallel at the expense of token-level sequential fidelity (Wu et al., 17 Jun 2026). A plausible implication is that the curriculum functions less as a generic optimization heuristic and more as a constraint-preserving mechanism that maintains local causal priors while broadening the admissible inference granularity.

4. Inference procedures and decoding regimes

At deployment, inference block size is chosen to trade off efficiency and accuracy, and the model generalizes across $B$4 (Wu et al., 17 Jun 2026). The principal diffusion decoding method is LowConfidence decoding. For the current block, tokens whose confidence exceeds a threshold $B$5 are committed, while the rest remain masked and are reconsidered in subsequent denoising passes (Wu et al., 17 Jun 2026). Temperature is 0 for diffusion baselines under thresholded decoding (Wu et al., 17 Jun 2026).

The per-block diffusion procedure is specified as follows (Wu et al., 17 Jun 2026):

1. Initialize $B$6 as fully masked and keep $B$7 fixed.
2. Run a forward pass to obtain token-wise confidences within $B$8.
3. Commit tokens meeting the criterion: $B$9 for LowConfidence, or $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$0 for RelaxedConfidence.
4. Repeat until all tokens in $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$1 are committed, then move to block $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$2.

The paper also evaluates a strict autoregressive baseline in which only the leftmost masked token is committed per step (Wu et al., 17 Jun 2026). This is not the main intended operating mode, but it serves as an analytical control for measuring the effect of local causal ordering.

A further decoding variant, RelaxedConfidence, is described as an analytical probe to accelerate block-wise decoding (Wu et al., 17 Jun 2026). For masked position $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$3, with neighborhood $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$4 and reliable set $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$5, the position-specific support and threshold are

$$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$6

The reported settings are $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$7, $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$8, $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0)$$9, and $b^k$0 (Wu et al., 17 Jun 2026). Tokens with stronger local support can therefore be committed earlier at a lower effective threshold (Wu et al., 17 Jun 2026).

5. Empirical performance

DreamReasoner-8B is evaluated on mathematical and code reasoning benchmarks and is reported as competitive with leading open autoregressive models such as Qwen3-8B (Wu et al., 17 Jun 2026). For the base model, the paper reports the following scores: MMLU 75.4, ARC-E 87.0, ARC-C 63.5, HellaSwag 76.6, PIQA 81.2, WinoGrande 74.2, RACE 43.8, GSM8K 83.4, MATH 55.8, GPQA 42.9, HumanEval 69.5, and MBPP 71.7 (Wu et al., 17 Jun 2026). DreamReasoner-8B-Base outperforms full-sequence diffusion baselines such as Dream-v0-7B-Base and LLaDA-8B-Base and is competitive with autoregressive base models such as Qwen3-8B-Base (Wu et al., 17 Jun 2026).

The continual pretraining ablation from Qwen3-8B-Base isolates the value of block diffusion. The reported scores are shown below.

Model	GSM8K	MATH	HumanEval	MBPP
Qwen3-8B-Base	86.5	52.7	65.8	68.8
DreamReasoner-8B-Base	83.5	51.2	62.8	60.4
w/o block diffusion	75.5	40.6	31.1	41.6

These results indicate that block diffusion substantially outperforms full-sequence diffusion for reasoning-intensive tasks (Wu et al., 17 Jun 2026).

The pilot study directly measures the interaction of training block size, inference block size, and decoding strategy. Train-$b^k$1 models remain comparatively robust across inference granularities, while train-$b^k$2 models degrade sharply under diffusion decoding on complex tasks (Wu et al., 17 Jun 2026). For example, Train $b^k$3 Infer $b^k$4 yields AIME25 21.3 under LowConfidence, whereas Curriculum $b^k$5 Infer $b^k$6 yields 38.3 (Wu et al., 17 Jun 2026). Likewise, Curriculum $b^k$7 Infer $b^k$8 gives AIME25 43.8 under LowConfidence (Wu et al., 17 Jun 2026).

For the final DreamReasoner-8B under PromptCoT 2.0 SFT with LowConfidence $b^k$9, the reported results across inference block sizes are:

Inference block size	AIME24	AIME25	LiveCodeBench
$b^{<k}_0$0	73.8	65.0	51.3
$b^{<k}_0$1	71.7	64.6	53.9
$b^{<k}_0$2	71.7	62.9	53.6
$b^{<k}_0$3	68.3	63.3	50.4

The paper compares these to Qwen3-8B-Thinking at 76.0, 67.3, and 52.8 on AIME24, AIME25, and LiveCodeBench respectively (Wu et al., 17 Jun 2026). It also reports stronger performance than several other open 7B–8B autoregressive baselines, including DeepSeek-R1-Distill-Qwen-7B, MiMo-7B-RL, and AceReason-Nemotron-1.1-7B (Wu et al., 17 Jun 2026). Relative to diffusion baselines, DreamReasoner-8B is reported to generalize across inference block sizes while SDAR-30B-A3B-Sci collapses as block size grows (Wu et al., 17 Jun 2026).

6. Efficiency characteristics

The efficiency analysis centers on tokens-per-forward-pass (TPF), defined as generated tokens divided by total forward passes, including token-prediction passes and any KV-cache recomputation (Wu et al., 17 Jun 2026). Relative to autoregression, block-wise parallel denoising increases TPF because multiple positions can be committed within one denoising pass (Wu et al., 17 Jun 2026).

The paper reports that TPF increases monotonically with larger inference block sizes while accuracy remains stable for DreamReasoner-8B (Wu et al., 17 Jun 2026). For $b^{<k}_0$4 and LowConfidence $b^{<k}_0$5, TPF rises across the response from about 1.2–1.6 in early blocks to about 2.8–3.5 in later blocks as more context accumulates (Wu et al., 17 Jun 2026). Threshold sensitivity is nontrivial: lower $b^{<k}_0$6 increases TPF but risks accepting low-confidence tokens, and code is particularly sensitive to aggressive thresholds (Wu et al., 17 Jun 2026).

RelaxedConfidence improves throughput without material accuracy loss according to the reported measurements (Wu et al., 17 Jun 2026). During thinking, AIME 2025 TPF increases from about 2.1 to about 2.8, and LiveCodeBench from about 1.7 to about 1.9 (Wu et al., 17 Jun 2026). During answering, AIME 2025 rises from about 2.4 to about 3.6, and LiveCodeBench from about 3.9 to about 6.2 (Wu et al., 17 Jun 2026). Earlier summary statistics in the paper describe average TPF gains of 22.5% and 54.5% for thinking and answering phases under relaxed thresholds (Wu et al., 17 Jun 2026).

The paper does not report wall-clock times (Wu et al., 17 Jun 2026). Accordingly, its efficiency claims are framed in terms of decoding granularity and TPF rather than end-to-end latency. This suggests that DreamReasoner-8B’s practical advantage depends not only on model quality but also on implementation details in Megatron-LM, FlexAttention, and SGLang.

7. Interpretation, limitations, and relation to broader research

The qualitative analysis attributes the failure of large-block-only training to overfitting toward intra-block parallel aggregation, which compromises token-level sequential reasoning and makes diffusion decoding brittle on complex tasks (Wu et al., 17 Jun 2026). The fact that strict autoregressive decoding partially recovers performance in those models is used to argue that local causal dependencies remain critical for long-form reasoning even in a diffusion setting (Wu et al., 17 Jun 2026). The curriculum is therefore presented as a way to preserve fine-grained local causality while adding robustness to coarse-grained, parallel generation (Wu et al., 17 Jun 2026).

Several limitations are explicit. Content-adaptive block partitioning is not explored, though the paper identifies it as a possible route to improving the efficiency-quality frontier (Wu et al., 17 Jun 2026). Evaluation is concentrated on math and code rather than broader domains such as tool use and agentic coding (Wu et al., 17 Jun 2026). Decoding remains bottlenecked by conservative per-token thresholds, and the paper identifies principled accept/reject mechanisms and stability analyses for larger models and longer contexts as open questions (Wu et al., 17 Jun 2026).

Recommended usage settings are also domain-specific. For math CoT tasks, $b^{<k}_0$7 is reported to balance accuracy and parallelism, with $b^{<k}_0$8 for conservative decoding; for latency-critical settings, $b^{<k}_0$9 with $\begin{equation} \begin{split} \mathcal{L}(\theta) \!=\! -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \!\mid\! b^{<k}_0\!,b^k_t\bigr)\biggr]\!, \end{split} \label{eq:block-loss} \end{equation}$0 is suggested, with only modest accuracy drop for DreamReasoner-8B (Wu et al., 17 Jun 2026). For code generation, $\begin{equation} \begin{split} \mathcal{L}(\theta) \!=\! -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \!\mid\! b^{<k}_0\!,b^k_t\bigr)\biggr]\!, \end{split} \label{eq:block-loss} \end{equation}$1 with $\begin{equation} \begin{split} \mathcal{L}(\theta) \!=\! -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \!\mid\! b^{<k}_0\!,b^k_t\bigr)\biggr]\!, \end{split} \label{eq:block-loss} \end{equation}$2 is recommended, and aggressive threshold reductions are discouraged unless RelaxedConfidence is used (Wu et al., 17 Jun 2026).

In the landscape of reasoning models, DreamReasoner-8B is significant because it provides a concrete, open-source recipe for reasoning-capable diffusion language modeling at the 8B scale (Wu et al., 17 Jun 2026). Its main technical thesis is not merely that diffusion can be applied to reasoning, but that the training granularity of denoising must be aligned with the causal structure required by long-CoT generation. A plausible implication is that future diffusion reasoners may be differentiated less by raw parameter scale than by the degree to which their training curriculum, block structure, and commit criteria preserve sequential fidelity under parallel decoding.