T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (2501.11651v2)

Abstract: LLMs have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through oversampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification.

• The paper introduces T1, a novel framework that uses synthesized chain-of-thought data and scaled reinforcement learning to improve language model reasoning and enable inference scaling.
• T1 achieves significant performance gains on reasoning benchmarks like MATH500 and exhibits inference scaling as compute budget increases.
• This work suggests broader application of RL for enhancing LLMs and offers insights into optimizing architectures and training via inference scaling.

Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling

The manuscript titled "Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling" presents a comprehensive exploration into leveraging reinforcement learning (RL) to enhance the reasoning capabilities of LLMs. It seeks to overcome the limitations of imitation learning, particularly in scaling inference capabilities during test-time. The proposed model, T1, is positioned to expand the field of LLM performance, especially concerning complex reasoning tasks such as mathematical problem solving.

The paper begins by identifying a significant gap in current LLM methodologies, noting that while LLMs have demonstrated remarkable reasoning abilities, the bulk of these advancements are attributable to imitation learning. This method falls short in facilitating effective scaling at test-time, as it doesn't allow the model to improve upon its own outputs through feedback and exploration—an area where RL holds potential.

T1 introduces a framework that employs synthesized chain-of-thought data for initial LLM fine-tuning, which amalgamates trial-and-error approaches with self-verification. This pre-training stage is critical in broadening the model’s exploration capabilities and preparing it for more intensive RL training stages. Furthermore, T1 encourages greater sampling diversity during RL training through oversampling strategies, thus promoting a more comprehensive exploration of potential reasoning pathways. Notably, the implementation of scaled RL training in T1 yields significant improvements over current models in competitive benchmarks such as MATH500, AIME2024, and Omni-MATH-500.

A distinctive feature of T1 is its ability to exhibit inference scaling behavior, wherein increasing inference budgets result in enhanced task performance even without additional result verification. This is accomplished through the separation of intermediate reasoning step generation from final answer derivation, allowing for increased flexibility and efficiency in reasoning tasks. For instance, T1 based on Qwen2.5-32B notably surpasses its predecessor Qwen QwQ-32B-Preview in reasoning benchmarks.

In examining the scalability of inference, the paper introduces a framework to scrutinize how inference length correlates with performance improvements. This involves truncating the reasoning process into varying lengths and studying the progression of accuracy enhancements as inference budgets grow. Such efforts indicate that exposing LLMs to increased computational budgets can lead to considerable improvements, aligning with theories of computational scaling in LLMs.

The implications of this research are multifaceted. Practically, the findings advocate for broader application of RL in refining LLMs, transitioning from a reliance on static, imitation-learned pathways to dynamic, feedback-driven improvements. Theoretically, the insights into inference scaling pave the way for future explorations in optimizing LLM architectures and training strategies, potentially leading to more robust and capable AI systems.

Future research directions indicated by this work include expanding the scope of T1’s application across diverse domains beyond mathematical reasoning. Additionally, further investigation into RL's scalability could yield novel strategies in reward modeling and policy optimization, offering richer insights into effective agent learning paradigms.

In conclusion, this paper marks a significant step in enhancing the reasoning capabilities of LLMs, utilizing RL and inference scaling as pivotal tools in this endeavor. By enabling self-correction and deeper exploration, T1 fosters not only more accurate reasoning among LLMs but also promotes the development of LLMs as more adaptive, self-improving entities in computational intelligence.