Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning (2502.06781v1)

Abstract: Reasoning abilities, especially those for solving complex math problems, are crucial components of general intelligence. Recent advances by proprietary companies, such as o-series models of OpenAI, have made remarkable progress on reasoning tasks. However, the complete technical details remain unrevealed, and the techniques that are believed certainly to be adopted are only reinforcement learning (RL) and the long chain of thoughts. This paper proposes a new RL framework, termed OREAL, to pursue the performance limit that can be achieved through \textbf{O}utcome \textbf{RE}w\textbf{A}rd-based reinforcement \textbf{L}earning for mathematical reasoning tasks, where only binary outcome rewards are easily accessible. We theoretically prove that behavior cloning on positive trajectories from best-of-N (BoN) sampling is sufficient to learn the KL-regularized optimal policy in binary feedback environments. This formulation further implies that the rewards of negative samples should be reshaped to ensure the gradient consistency between positive and negative samples. To alleviate the long-existing difficulties brought by sparse rewards in RL, which are even exacerbated by the partial correctness of the long chain of thought for reasoning tasks, we further apply a token-level reward model to sample important tokens in reasoning trajectories for learning. With OREAL, for the first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL, being on par with 32B models. OREAL-32B also surpasses previous 32B models trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our investigation also indicates the importance of initial policy models and training queries for RL. Code, models, and data will be released to benefit future research\footnote{https://github.com/InternLM/OREAL}.

• The paper introduces OREAL, a novel reinforcement learning framework utilizing binary outcome rewards to significantly enhance mathematical reasoning capabilities in large language models.
• OREAL achieves state-of-the-art performance, demonstrating 94.0 pass@1 accuracy on the challenging MATH-500 dataset with a 7B model, outperforming larger models.
• The research challenges traditional reinforcement learning views by proving that behavior cloning on positive samples suffices for optimal policy learning with binary feedback, offering a scalable approach.

Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning

The latest paper on enhancing the reasoning abilities of LLMs, particularly in solving mathematical problems, proposes a new reinforcement learning (RL) paradigm named OREAL (Outcome REwArd-based reinforcement Learning). The paper addresses critical challenges in refining artificial intelligence's approach to mathematical reasoning, specifically by optimizing the utilization of binary outcome rewards in RL processes. Given the rising significance of proprietary model advancements like OpenAI's series models, this research seeks to elucidate methodologies that can drive LLMs to reach the theoretical performance limits in complex reasoning tasks.

Key Contributions

1. OREAL Framework: The paper introduces the OREAL framework, uniquely designed to tackle mathematical reasoning tasks using RL driven by binary outcome rewards. This approach leverages the inherent structure of mathematical problems, focusing on the applicability of binary correctness feedback as a reliable reward signal.
2. Behavior Cloning and KL Regularization: A prominent theoretical contribution is the proof that behavior cloning applied to positive trajectories obtained from Best-of-N (BoN) sampling suffices to derive the Kullback-Leibler (KL)-regularized optimal policy within environments providing binary feedback. This insight redefines traditional RL approaches by simplifying feedback interpretation and leveraging positive sample behaviors as the primary learning trajectory.
3. Reward Shaping for Consistent Gradients: Addressing the challenges posed by sparse reward distributions and the partial correctness in reasoning chains, the paper suggests reshaping the rewards of negative samples to ensure gradient consistency between positive and negative samples. This method facilitates more stable optimization, accommodating both successful and unsuccessful reasoning trajectories.
4. Token-Level Reward Model: To further mitigate sparse reward issues, the paper implements a token-level reward model. This model enables finer granularity in the reward distribution across reasoning steps, highlighting critical tokens that contribute disproportionately to achieving a correct solution.

Experimental Validation

The OREAL framework demonstrated its effectiveness by achieving unprecedented results across various benchmark datasets. Notably, a 7B parameter model using OREAL attained a 94.0 pass@1 accuracy on the MATH-500 dataset, challenging the performance line held by notably larger 32B models. Moreover, OREAL-32B surpassed previous 32B models trained via distillation, yielding a pass@1 accuracy of 95.0.

Implications

Practical Implications: The proposed framework offers a scalable approach to enhance the reasoning capabilities of LLMs beyond traditional boundaries set by distillation-based learning paradigms. By focusing on outcome-driven learning methodologies, OREAL provides a viable alternative in domains that suffer from limited dataset availability or text data with sparse feedback.

Theoretical Implications: The research challenges existing RL paradigms by suggesting that the conventional view of exhaustive sampling strategies could be reimagined. The ability to drive performant learning from binary outcome reward significantly enhances the theoretical underpinnings of optimizing RL processes in scenarios constrained by limited actionable feedback.

Future Directions

The research lays a foundation for future exploration into refining RL strategies in artificial general intelligence (AGI) domains, particularly those requiring intricate reasoning capabilities. Prospective advancements might include the integration of meta-learning strategies to dynamically adapt reward shaping mechanisms, or employing hierarchical RL to further dissect reasoning tasks into modular sub-components. Additionally, exploring the interplay between initial policy model selection and RL efficiency can yield further insights into tailoring learning trajectories conducive to various task-specific environments.

In summary, this paper provides a comprehensive and theoretically backed framework in OREAL that attests to the significant advancements possible in mathematical reasoning for AI through strategic RL applications, setting a new precedent for future methodologies in artificial intelligence development.