Block-Size Curriculum Learning

• The paper shows that gradually increasing block size during training preserves token-level causal reasoning while improving overall inference robustness.
• Block-Size Curriculum Learning is defined as a method that adjusts block granularity, starting with fine details and shifting to coarse denoising to balance accuracy and efficiency.
• Empirical results on benchmarks like AIME and MATH500 validate that a progressive block schedule maintains high performance across different inference settings.

Searching arXiv for the specified papers and closely related curriculum-learning work. Block-size curriculum learning is a curriculum-learning strategy in which the effective granularity of a learning problem is controlled by varying the size of contiguous blocks during training. In diffusion reasoning models, it denotes training with small blocks first and then gradually increasing block size so that fine-grained token-level causal dependencies are learned before the model is exposed to more parallel, coarser denoising regimes (Wu et al., 17 Jun 2026). In theoretical analyses of compositional generalization, the same principle appears as recursive decomposition of long computations into shorter blocks that can be learned and then composed, making block length an explicit proxy for problem difficulty and learnability (Rajaraman et al., 26 Jun 2026).

1. Formalization in block diffusion LLMs

The most explicit operational definition of block-size curriculum learning appears in block diffusion LLMs. In that setting, the clean sequence is partitioned into contiguous blocks,

$\mathbf{x}_0 = (b_0^1, b_0^2, \dots, b_0^K),$

where each block contains $B$ tokens, and generation is factorized autoregressively across blocks:

$$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$

with $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$. Each block is then denoised with a local discrete diffusion process, so that for the $k$-th block the model predicts

$p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$

Training uses a weighted cross-entropy on masked positions,

$$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$

where

$w_t \coloneqq \frac{\alpha_t'}{1-\alpha_t}.$

Within this formalism, block-size curriculum learning means starting training with small blocks and then gradually increasing the block size during training (Wu et al., 17 Jun 2026).

This definition is specific to a class of diffusion-based sequence models in which intra-block denoising is parallel but inter-block generation remains autoregressive. The curriculum is therefore not merely a schedule over data subsets; it is a schedule over the causal granularity at which the model must form and refine predictions.

2. Granularity gap and the rationale for curriculum

The primary motivation for block-size curriculum learning is a granularity gap between the regime that best preserves reasoning quality and the regime that maximizes decoding efficiency. Small blocks provide a stronger token-level causal scaffold, whereas large blocks expose more tokens to parallel denoising and therefore offer a more efficient inference regime. The trade-off is that large blocks require the model to maintain more long-range intra-block structure and fine-grained reasoning without the strict left-to-right discipline of autoregressive generation (Wu et al., 17 Jun 2026).

Empirically, naively training with large block sizes yields remarkably poor reasoning, while small block sizes preserve effective reasoning. The reported interpretation is that direct large-block training causes severe reasoning degradation, especially on long chain-of-thought tasks, because the model overfits to coarse intra-block aggregation and loses the ability to decompose reasoning into reliable fine-grained denoising steps (Wu et al., 17 Jun 2026).

A related theoretical rationale is developed in the study of semiautomaton simulation. There, the update rule is stationary, Markovian, and time-homogeneous,

$$s_t \gets \pi(s_{t-1}, w_t), \qquad \pi: S \times \Sigma \to S.$$

Because any contiguous chunk of a length-$T$ computation is itself a valid shorter computation, long-horizon reasoning can be reduced to shorter blocks and then composed. This yields a self-generated curriculum in which the learner adaptively decomposes hard instances into easier ones rather than relying on a fixed external difficulty schedule (Rajaraman et al., 26 Jun 2026).

Taken together, these results support a common interpretation: block size functions as a structural control variable over how much local causal guidance is available during learning. Small blocks privilege local precision; large blocks privilege parallelism. Curriculum learning is introduced to traverse that trade-off without incurring the failure modes of direct coarse-grained training.

3. Training schedules and implementation patterns

In DreamReasoner-8B, two concrete schedules are reported. The pilot curriculum is a two-stage progression $B$0: training lasts for 4 epochs total, with the first 3 epochs using block size 4 and the final 1 epoch using block size 32. The full DreamReasoner-8B schedule begins with block size 4 for 3 epochs and then switches to mixed-granularity training with block sizes randomly sampled from $B$1 (Wu et al., 17 Jun 2026).

Setting	Schedule	Reported purpose
Pilot curriculum	4 epochs total; first 3 epochs at block size 4; final 1 epoch at block size 32	Simple two-stage progression $B$2
DreamReasoner-8B	Block size 4 for 3 epochs; then mixed $B$3	Preserve small-block competence while adding coarse-grained robustness

The mixed-granularity final stage is not incidental. The authors state that keeping small blocks in the final stage is important because training only on large blocks with long reasoning traces hurts small-block inference (Wu et al., 17 Jun 2026). This directly distinguishes curriculum learning from a simple late-stage switch to the deployment block size.

The same paper connects training schedule to decoding strategy. Two decoding modes are compared: LowConfidence diffusion decoding, which unmasks tokens whose confidence exceeds $B$4, and autoregressive decoding, which reveals only the leftmost masked token. Curriculum training improves robustness under both decoding strategies, indicating that the schedule affects not only train–test block-size alignment but also the stability of the denoising dynamics themselves (Wu et al., 17 Jun 2026).

In the semiautomata setting, schedule design takes a recursive rather than epoch-based form. A composition operator

$B$5

chunks the input into $B$6 segments of length $B$7, applies $B$8 to each chunk, and feeds each predicted boundary state into the next chunk. The multiscale algorithm $B$9 iterates composition, boosting, and splitting across levels, thereby turning block size into a recursive horizon parameter rather than a single global hyperparameter (Rajaraman et al., 26 Jun 2026).

4. Empirical behavior in diffusion reasoning

The central empirical result is that curriculum training $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$0 produces better cross-granularity robustness than either fixed small-block or fixed large-block training. In a pilot study on AIME 2024, AIME 2025, and MATH500, training with block size 4 is strong and robust, training with block size 32 is brittle and much worse, and curriculum $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$1 recovers robustness across both inference block sizes 4 and 32 (Wu et al., 17 Jun 2026).

The disparity is especially sharp on long reasoning benchmarks. With training block size 4, LowConfidence diffusion decoding at inference block size 4 attains AIME 2024 $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$2, AIME 2025 $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$3, and MATH500 $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$4; with training block size 32, the corresponding values drop to $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$5, $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$6, and $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$7. Under the curriculum $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$8, the same inference setting yields $$p_{\theta}(\mathbf{x}_0) = \prod_{k=1}^{K} p_{\theta}(b^k_0 \mid b^{<k}_0),$$9, $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$0, and $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$1. At inference block size 32, the curriculum remains stable, with $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$2, $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$3, and $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$4, whereas direct large-block training remains substantially weaker on AIME 2025 and MATH500 (Wu et al., 17 Jun 2026).

The paper interprets these results as evidence that small-block training builds reliable local reasoning skills and that large-block inference becomes usable only after those skills have been established. This suggests that the curriculum decouples reasoning quality from block granularity, allowing larger inference blocks without the severe collapse seen in direct large-block training.

The final DreamReasoner-8B model extends this pattern to larger-scale evaluation. Reported results across inference block sizes 4, 8, 16, and 32 are: AIME 2024 $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$5; AIME 2025 $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$6; and LiveCodeBench $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$7. A comparison point given in the paper is SDAR-30B-A3B-Sci, whose block-4 and block-32 results are AIME 2024 $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$8 and $b^{<k}_0 = (b^1_0,\dots,b^{k-1}_0)$9, AIME 2025 $k$0 and $k$1, and LiveCodeBench $k$2 and $k$3 (Wu et al., 17 Jun 2026).

The reported qualitative conclusion is that the curriculum-trained model preserves performance as inference blocks get larger, unlike prior diffusion models whose accuracy collapses with larger blocks. This stability is particularly emphasized on code benchmarks, where DreamReasoner-8B is described as especially strong relative to the diffusion baselines considered in the study (Wu et al., 17 Jun 2026).

5. Theoretical treatment through compositional generalization

Theoretical work on compositional generalization provides a second, more abstract formulation of block-size curriculum learning. In semiautomaton simulation, the learner must predict the terminal state after $k$4 transitions. Rather than solving the length-$k$5 problem directly, the curriculum-based approach recursively decomposes longer sequences into shorter subproblems, learns to solve those, and composes the solutions (Rajaraman et al., 26 Jun 2026).

In the interactive SFT setting, the learner may query an oracle for intermediate states. A naive full-chain-of-thought strategy labels every step of every length-$k$6 instance and therefore incurs an $k$7 query burden per example. The curriculum algorithm instead splits a length-$k$8 instance into shorter blocks, queries boundary states, trains a predictor on the shorter subproblems, and composes the learned short-range model across blocks. The resulting complexity bound is subpolynomial:

$k$9

This overcomes the direct simulation barrier $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$0 for full-chain-of-thought learners (Rajaraman et al., 26 Jun 2026).

A depth-1 warmup makes the length-versus-accuracy trade-off explicit. If the computation is partitioned into $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$1 blocks of length $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$2, and a model attains error

$p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$3

on length-$p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$4 blocks, then $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$5-fold composition yields error at most $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$6 on length-$p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$7 instances. The paper then generalizes this idea through a multiscale recursion $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$8, whose levels perform composition, boosting on reweighted hard subproblems, and splitting via boundary labels (Rajaraman et al., 26 Jun 2026).

The RLVR analysis makes the block-size claim even more directly. Standard RLVR succeeds when a reference model has nontrivial sequence-level coverage at the full target length $p_\theta(x^{k,i}_0 \mid b^{<k}_0, b^k_t).$9. The paper instead assumes only block-level competence at a shorter block length $$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$0, then uses a Guess-and-Check procedure with a verifier to simulate intermediate labels and recover boundary states block by block. The main claim is that curriculum reduces the requirement from coverage at length $$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$1 to coverage at block length $$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$2, which the paper characterizes as an exponentially weaker condition (Rajaraman et al., 26 Jun 2026).

The block-level learner $$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$3 is proved to require sample complexity

$$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$4

with verifier query complexity

$$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$5

and computational cost

$$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$6

For the full length-$$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$7 problem, the paper states a leading sample complexity

$$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$8

with additional query and compute terms scaling with $$\mathcal{L}(\theta) = -\mathbb{E}_{t,b_0,b_t}\biggl[\sum_{k=1}^{K}w_t \log p_{\theta}\bigl(b^{k}_0 \mid b^{<k}_0,b^k_t\bigr)\biggr],$$9 rather than with full-length coverage assumptions (Rajaraman et al., 26 Jun 2026). The theoretical significance is that competence at a short horizon can be expanded compositionally to a long horizon.

6. Relation to other curriculum variables, misconceptions, and limitations

Block-size curriculum learning belongs to a broader family of curricula in which a structural property of the task is used as a difficulty signal. In few-shot meta-learning, for example, the number of labeled examples per class in the support set is treated as a natural proxy for task difficulty: larger support size $w_t \coloneqq \frac{\alpha_t'}{1-\alpha_t}.$0 yields lower-variance inner-loop gradients and easier tasks, while progressively reducing $w_t \coloneqq \frac{\alpha_t'}{1-\alpha_t}.$1 moves training toward the true few-shot test regime. The schedule begins at $w_t \coloneqq \frac{\alpha_t'}{1-\alpha_t}.$2 and decreases stagewise until $w_t \coloneqq \frac{\alpha_t'}{1-\alpha_t}.$3, with the final half of training allocated to the target support size (Stergiadis et al., 2021).

This comparison is not an identity of methods, because support-set curriculum and block-size curriculum modify different objects. The former changes the amount of supervision available in each task, whereas the latter changes the granularity of generation or decomposition. A plausible implication is that both instantiate a common design pattern: curriculum over structural granularity rather than over semantic content.

Several misconceptions are directly addressed by the reported ablations. First, curriculum learning is not equivalent to simply training on large blocks or large supports. In diffusion reasoning, direct large-block training is markedly worse than a progression from small to large blocks (Wu et al., 17 Jun 2026). In few-shot meta-learning, static large-support training does not match the curriculum versions and can even underperform baseline MAML, indicating that the gain comes from the progressive reduction of support size rather than from using a larger support set at all times (Stergiadis et al., 2021).

Second, block-size curriculum learning is not reducible to train–test matching alone. In DreamReasoner-8B, the final schedule retains small blocks in the mixed-granularity stage specifically because exclusive large-block training harms small-block inference (Wu et al., 17 Jun 2026). This indicates that robustness across granularities, not just adaptation to a single deployment block size, is part of the objective.

The current limitations are explicit. DreamReasoner-8B does not explore dynamic or content-aware block partitioning, such as variable-length blocks aligned with syntax or semantics; it focuses mostly on mathematics and code generation rather than systematically evaluating tool use or agentic tasks; and decoding efficiency remains constrained by conservative commitment thresholds, motivating the separate RelaxedConfidence method (Wu et al., 17 Jun 2026). The compositional generalization analysis, for its part, is developed on the canonical problem of semiautomata simulation rather than on unrestricted natural-language reasoning (Rajaraman et al., 26 Jun 2026).

The present state of the topic therefore supports a narrow but clear conclusion. Block-size curriculum learning is a principled method for traversing a granularity trade-off: begin with short blocks that preserve local causal structure, then enlarge or compose those blocks so that the model can operate efficiently at longer horizons or coarser denoising resolutions. In empirical diffusion reasoning, this strategy stabilizes performance across inference block sizes; in theory, it reduces long-horizon learning to the acquisition and composition of shorter-horizon competence (Wu et al., 17 Jun 2026, Rajaraman et al., 26 Jun 2026).