Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning (2402.05808v2)

Abstract: In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for LLMs. The core challenge in applying RL to complex reasoning is to identify a sequence of actions that result in positive rewards and provide appropriate supervision for optimization. Outcome supervision provides sparse rewards for final results without identifying error locations, whereas process supervision offers step-wise rewards but requires extensive manual annotation. R$^3$ overcomes these limitations by learning from correct demonstrations. Specifically, R$^3$ progressively slides the start state of reasoning from a demonstration's end to its beginning, facilitating easier model exploration at all stages. Thus, R$^3$ establishes a step-wise curriculum, allowing outcome supervision to offer step-level signals and precisely pinpoint errors. Using Llama2-7B, our method surpasses RL baseline on eight reasoning tasks by $4.1$ points on average. Notebaly, in program-based reasoning on GSM8K, it exceeds the baseline by $4.2$ points across three backbone models, and without any extra data, Codellama-7B + R$^3$ performs comparable to larger models or closed-source models.

• The paper introduces Reverse Curriculum Reinforcement Learning, a dynamic programming method that trains LLMs by starting from the end of correct demonstrations to simplify complex tasks.
• The paper achieves an average improvement of 4.1 points across eight reasoning tasks, with a 4.2-point boost on GSM8K, demonstrating superior performance over traditional RL approaches.
• The paper highlights RCRL's potential to reduce reliance on dense supervision, lowering annotation costs and enhancing scalability for training advanced reasoning models.

Analyzing "Training LLMs for Reasoning through Reverse Curriculum Reinforcement Learning"

The paper, "Training LLMs for Reasoning through Reverse Curriculum Reinforcement Learning," presents an innovative approach for enhancing the reasoning capabilities of LLMs using a method coined as Reverse Curriculum Reinforcement Learning (RCRL). Traditionally, the application of reinforcement learning (RL) in the context of complex reasoning tasks has been hindered by the choice between sparse outcome supervision and dense, but expensive, process supervision. RCRL addresses this dichotomy by leveraging outcome supervision in a manner that mimics the benefits of step-wise process supervision.

Methodological Overview

RCRL stands out by employing a reverse curriculum strategy where the LLM is trained to solve problems by beginning the learning process from the end of correct demonstrations and gradually moving towards initial states. This technique allows the model to engage with simpler sub-problems initially, which are incrementally complexified. Such an approach reduces the exploratory burden on the model and fosters more efficient error correction and learning.

The authors articulate their approach as a dynamic programming solution where learning is linear in the number of reasoning steps, mitigating the usual exponential complexity growth associated with traditional RL approaches in LLM reasoning tasks. This methodology involves using proxal policy optimization (PPO), a technique that ensures stable model performance by incorporating KL divergence between the learned policy and the initialized policy.

Experimental Evaluation

The empirical validation of RCRL includes rigorous testing on diverse reasoning tasks such as mathematical problem solving, logical reasoning, reading comprehension, and natural language inference (NLI). The model, utilizing Llama2-7B as a backbone, achieves an average improvement of 4.1 points over RL baselines across eight reasoning tasks. Notably, in the GSM8K program-based reasoning task, it demonstrates superiority by a margin of 4.2 points over traditional RL methods.

Implications and Future Directions

The principal contribution of this research lies in its ability to harness the benefits of step-level supervision without the associated annotation overhead, which is significant in scaling the reasoning capabilities of LLMs. The results suggest that RCRL could serve as a potent catalyst in developing models that require less human-curated data, thereby reducing operational costs and increasing accessibility.

The findings open new pathways for further exploration, such as integrating RCRL with other forms of reasoning verification techniques to improve LLM performance further or applying this methodology to even larger model architectures to test scalability. The capability to extend this framework to more challenging and nuanced reasoning contexts, beyond mathematical and logic-based problems, marks an intriguing prospect for AI research.

Conclusion

The Reverse Curriculum Reinforcement Learning framework elucidated by Xi et al. provides a compelling paradigm for training LLMs in reasoning tasks. It successfully bridges the gap between the inadequacies of pure outcome supervision and the logistical complexities of process supervision. This approach holds promise for both theoretical advancements in machine learning frameworks and practical applications in educational and cognitive AI development. As such, it is poised to make a significant impact on the future landscape of AI reasoning systems.