Learning to Reason with Curriculum II: Compositional Generalization

Abstract: Compositional generalization, the ability to solve complex problems by combining solutions to simpler sub-problems, is a fundamental capability of both natural and artificial intelligence, and a key mechanism underlying chain-of-thought reasoning. However, the theoretical underpinnings of compositional generalization remain poorly understood: when and why does decomposing a problem into parts yield more efficient learning than solving it directly? We study this question through the canonical problem of learning to simulate semiautomata (predicting the outcome of $T$ steps of sequential computation), a model that captures state tracking, regular language recognition, and modular arithmetic. We show that an autocurriculum-based approach building on Part I of this series, recursively decomposing longer sequences into shorter sub-problems, learning to solve them, and composing the solutions, achieves dramatically better statistical complexity than direct methods. (i) For a setting inspired by supervised fine-tuning (SFT) where the learner receives interactive feedback on intermediate states of the computation, curriculum facilitates learning from only $2^{\mathcal{O}(\sqrt{\log T})}$ tokens of supervision; i.e., subpolynomial in the sequence length $T$, overcoming the $Ω(T)$ token barrier required by direct simulation. (ii) For a setting inspired by reinforcement learning with verifiable rewards (RLVR), where the learner improves a pre-trained reference model using an outcome verifier, we show that curriculum reduces the requirement on the reference model from coverage at the full sequence length $T$ to coverage at a shorter block length $B \ll T$, an exponentially weaker condition.

• The paper demonstrates that curriculum learning can decompose long-horizon tasks into manageable subproblems, significantly reducing sample and query complexity.
• It introduces the AutoLearn algorithm that recursively boosts weak learners via block-level composition and inverted sampling to address sequential challenges.
• The study establishes strong sublinear statistical guarantees in both supervised and RL settings, supporting scalable and efficient AI reasoning.

Compositional Generalization via Self-Generated Curriculum: A Theoretical Analysis

Problem Setting and Motivation

This work investigates compositional generalization—the ability for a learner to solve long-horizon sequential tasks by composing solutions to subproblems—via the lens of semiautomaton simulation. Semiautomata are Markovian sequential systems generalizing regular languages, state tracking, and modular arithmetic, providing an analytically tractable testbed for understanding reasoning and curriculum learning. The paper focuses on the question: Under what conditions, and with what techniques, can curriculum learning yield substantial improvements in sample and query complexities over standard approaches?

Two learning settings are formalized:

• Interactive Chain-of-Thought Supervision (iSFT): The learner adaptively queries intermediate states (not merely start and end states) within trajectories at unit cost per query.
• RL with Verifiable Rewards (RLVR): The learner improves a weak reference model, using only final-state verifier feedback, with coverage assumptions holding only for short blocks.

The central technical question is whether curriculum strategies exploiting compositional structure (i.e., decomposing longer sequential problems into curriculum-composed shorter blocks) permit dramatic reductions in query/sample complexity over naive (non-curriculum) supervised or RL fine-tuning.

Main Theoretical Contributions

1. Markovian Compositionality and Curriculum

By exploiting the Markovian property of semiautomata, the auto-curriculum training procedure recursively decomposes long-horizon tasks into shorter blocks. Each subproblem can be mastered with lower supervision cost, and the resulting submodels are composed into a high-accuracy predictor for the original task, albeit with careful control of error accumulation. Leveraging this structure:

• The problem of simulating a length-$T$ process is reduced to learning to simulate length-$B$ compositions and performing hierarchical boosting and model composition.

2. Algorithmic Construction

The principal algorithm is AutoLearn, which implements a multistage, recursive boosting-plus-composition strategy:

• At each recursion, longer sequences are partitioned into $L$ shorter blocks.
• Weak learners (constant-accuracy predictors per block) are aggregated (boosting), and their predictions composed over blocks, with repeated recursion until lowest-level subproblems can be learned via standard empirical risk minimization (e.g., next-token prediction).

Critical to the efficiency of this approach is a tailored "inverted sampling" routine for efficiently generating the correct distribution over sub-blocks necessary for boosting without the sample blowup inherent in rejection sampling.

3. Strong Statistical Guarantees

Supervised (iSFT) Setting:

AutoLearn provably achieves subpolynomial query and sample complexity in sequence length $T$, specifically:

• $$n_{\mathrm{sample}}, n_{\mathrm{query}} \leq 2^{O(\sqrt{\log T})} \cdot d\log(|S|) \log\frac{1}{\epsilon}\log\frac{1}{\delta} / \epsilon$$ where $d$ is the Natarajan dimension of the hypothesis class.

This statistic is strongly sublinear in $T$, breaking the $\Omega(T)$ lower bound for full-sequence SFT or end-to-end baselines. Lower bounds are also established: any naive SFT or end-to-end approach requires at least $\Omega(dT)$ queries to achieve comparable accuracy.

RL-with-Verifiable-Reward (RLVR) Setting:

Curriculum composition (AutoLearn.RL) allows the learner to bootstrap coverage:

• Only block-level coverage ($\mathcal{O}(B)$, $B$0) by the pretrained model is needed; full-sequence ($B$1) coverage is not required.
• The overall sample, query, and computational complexities depend polynomially on $B$2 (not $B$3), thus enabling generalization to long horizons under only local competence:

$B$4

with coverage-related "burn-in" terms only at the block level.

4. Technical Machinery

The analysis hinges on new sampling and potential function arguments:

• Inverted Sampling: Efficiently produces examples for boosting even as distributional targets shift due to recursive curriculum decomposition, avoiding the $B$5 cost (in number of blocks $B$6) of naive rejection sampling.
• Recursive Error Control: Carefully tracks error amplification through composition; a multiscale boosting structure is required to maintain a sub-linear sample/query cost.
• Sharp Lower Bounds: Full-chain SFT and end-to-end regularization are shown to be fundamentally limited, even computationally hard for some model classes.

5. Instantiations and Applicability

The bounds are instantiated for learning regular languages via finite automata and for linear recurrences over finite fields. In both cases, the guarantees scale only with representation complexity (DFA state count, or dimension for recurrences), not the exponential number of possible states, and only polylogarithmically (or subpolynomially) in horizon $B$7.

Implications for AI Reasoning and Future Directions

Practical Impact:

The results provide theoretical ground for the empirical success of curriculum learning and Self-Play/Learning-to-Reason paradigms in LLMs. They deliver provable support for length generalization and skill composition via block-wise curriculum—even in the presence of weak base models.

Theoretical Insights:

• Coverage Expansion: It is possible to leverage local correctness to expand coverage exponentially via compositional auto-curricula.
• Sample Complexity Barriers: Substantially sub-linear-in-$B$8 sample/query complexity can be achieved in structured settings—even when non-curriculum learning is linear or exponential in $B$9.
• Limits of Compositionality: Lower bounds show that for sufficiently unstructured (non-compositional) classes or non-Markovian/recurrent transitions, these gains may vanish.

Future Research:

• Beyond Markov: Extending results to general autoregressive models (transformers/LSTMs), where historical dependence or stochasticity breaks Markovian assumptions.
• Beyond Deterministic: Handling stochastic automata, including those where the true process is a Markov chain with state uncertainty or ambiguous outputs.
• Optimality: Tightening the dependency on $L$0 in query/sample complexity, possibly achieving polylogarithmic rates in richer model classes (e.g., those with finite Littlestone dimension).

Conclusion

This work advances a rigorous, provable framework for compositional generalization in sequential reasoning. Through the careful design of recursive curricula aligned with the compositional (Markovian) structure of the task, the authors demonstrate superpolynomial reductions in learning cost for both supervised and RL fine-tuning protocols, compared to traditional baselines. These results substantiate the empirical strategy of building LLM reasoning capabilities through self-generated curricula and composition, and provide compelling theoretical justification for scaling structured skills from local to global via curriculum. The extensions to more general architectures (e.g., transformers) and stochastic domains remain an important next step for the theory of reasoning in AI.