h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning (2510.07312v1)

Abstract: LLMs excel at short-horizon reasoning tasks, but performance drops as reasoning horizon lengths increase. Existing approaches to combat this rely on inference-time scaffolding or costly step-level supervision, neither of which scales easily. In this work, we introduce a scalable method to bootstrap long-horizon reasoning capabilities using only existing, abundant short-horizon data. Our approach synthetically composes simple problems into complex, multi-step dependency chains of arbitrary length. We train models on this data using outcome-only rewards under a curriculum that automatically increases in complexity, allowing RL training to be scaled much further without saturating. Empirically, our method generalizes remarkably well: curriculum training on composed 6th-grade level math problems (GSM8K) boosts accuracy on longer, competition-level benchmarks (GSM-Symbolic, MATH-500, AIME) by up to 2.06x. Importantly, our long-horizon improvements are significantly higher than baselines even at high pass@k, showing that models can learn new reasoning paths under RL. Theoretically, we show that curriculum RL with outcome rewards achieves an exponential improvement in sample complexity over full-horizon training, providing training signal comparable to dense supervision. h1 therefore introduces an efficient path towards scaling RL for long-horizon problems using only existing data.

• The paper introduces the h1 method that composes short-horizon tasks into long-horizon reasoning chains using outcome-only RL and a curriculum approach.
• It leverages synthetic data and stage-wise training, achieving substantial in-domain accuracy improvements (e.g., 175% gain at h=5) and acquiring new reasoning skills.
• It shows strong out-of-domain generalization to benchmarks and provides a scalable framework for enhanced long-horizon capabilities without extra annotations.

Bootstrapping Long-Horizon Reasoning in LLMs via Curriculum RL on Composed Short-Horizon Data

Introduction and Motivation

LLMs have demonstrated strong performance on short-horizon reasoning tasks, but their capabilities degrade rapidly as the required reasoning horizon increases. This limitation is particularly acute in domains such as mathematics, code synthesis, and scientific discovery, where multi-step, stateful reasoning is essential. The paper introduces a method, h1, that leverages abundant short-horizon data to synthetically construct long-horizon reasoning tasks, enabling scalable reinforcement learning (RL) with outcome-only rewards and a curriculum over horizon lengths. This approach circumvents the need for costly step-level supervision or inference-time scaffolding, providing a practical path to scaling LHR capabilities.

Figure 1: The h1 method composes short-horizon reasoning problems into longer sequences and applies a stage-wise curriculum to scale RL training, yielding improved long-horizon reasoning and OOD generalization.

Methodology: Compositional Data and Curriculum RL

Synthetic Long-Horizon Data Construction

The core of the method is the serial composition of atomic, verifiable short-horizon tasks (e.g., GSM8K math problems) into chains of arbitrary length. Each sub-problem’s input is a deterministic function of the previous sub-problem’s output, enforced via lightweight adapters (e.g., arithmetic transformations, unit conversions). The resulting prompt presents a sequence of interdependent sub-problems, with only the final answer used for supervision (outcome-only RL). This construction exposes the model to dependency chains requiring state tracking and intermediate value management, which are not exercised by atomic tasks alone.

Curriculum RL over Horizons

Training proceeds in a stage-wise curriculum: at each stage, the model is trained on chains of a fixed horizon length $h$, starting from $h=1$ and incrementally increasing to $H_{\max}$. The RL objective is optimized using Dr. GRPO, a variant of Group-Relative Policy Optimization, with outcome-only rewards. Parameters are carried forward between stages, allowing the model to first master short-horizon primitives and then progressively acquire horizon-dependent skills ($\sigma_j$) necessary for longer chains.

Baseline policies include:

• Only-L1: RL on $h=1$ only.
• Uniform-Mix: Uniform sampling over all horizons.
• Only-Long: RL on $h=H_{\max}$ only.

The curriculum is shown to be essential for effective LHR training, as direct training on long horizons suffers from vanishing reward signal and unstable gradients.

Empirical Results: In-Domain and Out-of-Domain Gains

In-Domain Performance

Curriculum RL on composed GSM8K chains yields substantial monotonic improvements in accuracy as the training horizon increases. For example, at $h=5$, accuracy improves from 3.57% (instruct model) to 9.82% (curriculum RL), a 175% relative gain. Baselines such as Only-L1 and Uniform-Mix fail to improve long-horizon performance, confirming that single-step accuracy is insufficient for LHR.

Figure 2: Curriculum RL on compositional data yields significant in-domain long-horizon reasoning gains (up to $2.9\times$), preventing RL saturation and requiring no new data.

Learning New Capabilities

A key empirical finding is that curriculum RL on composed data enables the model to acquire genuinely new reasoning skills, as evidenced by improvements at high pass@k (e.g., pass@128) that exceed the base model’s capabilities. This contradicts prior work showing that RLVR typically only improves sample efficiency for existing skills (Yue et al., 18 Apr 2025).

Figure 3: Curriculum RL on composed synthetic data outperforms RLVR on standard data even at pass@128, indicating the acquisition of new reasoning capabilities beyond single-step improvements.

Out-of-Domain Generalization

Models trained with the h1 curriculum on GSM8K chains generalize to significantly harder, implicit-horizon benchmarks such as MATH-500, AIME, and GSM-Symbolic. For instance, AIME 2024 accuracy improves from 5.10% to 10.52% (2.1× gain), and ultra-long-context benchmarks (LongBench-v2, Hash-hop) also show consistent improvements.

Figure 4: Long-horizon training on GSM8K generalizes to harder tasks, with AIME 2024 performance improving by $2.1\times$ and ultra-long-context capabilities by $1.2\times$.

Theoretical Analysis: Sample Complexity and Curriculum Efficiency

The paper provides a formal analysis of the sample complexity of curriculum RL versus direct full-horizon training. In a simplified model, direct outcome-only RL at horizon $H$ requires a batch size exponential in $H$ due to the vanishing probability of correct rollouts. In contrast, curriculum RL reduces the required batch size to polynomial in $H$ by ensuring that each stage operates at a horizon where the model’s success probability is non-negligible. This efficiency is shown to be equivalent to training with dense, per-step rewards, but without requiring step-level supervision.

Data-Compute Tradeoffs and Cost-Efficient Curricula

The method is robust to data constraints: high long-horizon performance can be achieved even when the training data distribution is skewed towards shorter horizons, provided sufficient training compute is allocated. This is critical for real-world scenarios where long-horizon data is scarce or expensive.

Figure 5: Under mild skew towards shorter samples, models can match uniform-sample baselines, though at increased training cost; low-cost data distributions can achieve near-optimal performance.

Scaling Laws and Distributional Search

Extensive experiments on synthetic arithmetic tasks confirm that, for a fixed compute budget, there exists a set of optimal data distributions that minimize data cost while achieving target accuracy. The feasible region for cost-efficient curricula is convex and can be found via simplex search.

Figure 6: Training distributions parameterized by a cost scalar, plotted against FLOP budget, reveal optimal tradeoffs between data cost and compute.

Figure 7: Training trajectories with 3-bin data distributions, visualized over the probability simplex, highlight the convex feasible region for cost-efficient curricula.

Qualitative Analysis: State Tracking and Error Modes

Qualitative examples demonstrate that untrained models exhibit state-tracking and substitution errors in long-horizon chains, while curriculum-trained models maintain correct intermediate values and logical consistency across steps, solving complex multi-step problems that require persistent state management.

Implications and Future Directions

The h1 method provides a scalable, data-efficient framework for bootstrapping LHR in LLMs using only existing short-horizon data. The approach is model-agnostic and generalizes across domains and architectures. Theoretical and empirical results indicate that curriculum RL on composed data can teach new reasoning skills, not merely refine existing ones, and that these skills transfer to harder, out-of-domain tasks.

Potential future directions include:

• Incorporating more diverse atomic skills beyond GSM8K.
• Exploring richer composition strategies (e.g., DAGs rather than chains) to further expand the space of long-horizon tasks.
• Integrating this approach with advances in long-context modeling and memory-augmented architectures.

Conclusion

This work demonstrates that LLMs’ long-horizon reasoning capabilities can be substantially improved by curriculum RL on synthetically composed short-horizon data, without requiring new annotations or step-level supervision. The method achieves strong in-domain and out-of-domain gains, imparts genuinely new reasoning skills, and is theoretically grounded in efficient sample complexity. These results have significant implications for scaling LHR in LLMs and for the design of future training curricula in AI systems.