OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning (2412.16849v1)

Abstract: OpenAI's recent introduction of Reinforcement Fine-Tuning (RFT) showcases the potential of reasoning foundation model and offers a new paradigm for fine-tuning beyond simple pattern imitation. This technical report presents \emph{OpenRFT}, our attempt to fine-tune generalist reasoning models for domain-specific tasks under the same settings as RFT. OpenRFT addresses two key challenges of lacking reasoning step data and the limited quantity of training samples, by leveraging the domain-specific samples in three ways: question augmentation, synthesizing reasoning-process data, and few-shot ICL. The evaluation is conducted on SciKnowEval, where OpenRFT achieves notable performance gains with only $100$ domain-specific samples for each task. More experimental results will be updated continuously in later versions. Source codes, datasets, and models are disclosed at: https://github.com/ADaM-BJTU/OpenRFT

• The paper introduces OpenRFT, which leverages reinforcement fine-tuning to adapt general reasoning models to specialized tasks with minimal data.
• It employs question augmentation and reasoning process synthesis to expand scarce domain-specific training samples and enhance model reliability.
• Empirical results on SciKnowEval show an 11% performance boost using only 100 samples per task, underlining its practical efficiency.

OpenRFT: Adapting Reasoning Foundation Models for Domain-specific Tasks with Reinforcement Fine-Tuning

The presented paper introduces a novel approach termed OpenRFT, designed to adapt generalist reasoning foundation models to specific domain tasks using Reinforcement Fine-Tuning (RFT). This method is a response to advancing reasoning models beyond pattern imitation, suggesting that RFT can effectively generalize them to diverse applications with limited domain-specific training samples. The work navigates the pivotal challenges of lacking reasoning step data and limited training samples by innovatively leveraging domain-specific data through question augmentation, reasoning process synthesis, and few-shot In-Context Learning (ICL).

Core Contributions and Methodologies

At its core, OpenRFT seeks to address critical issues inherent in fine-tuning models with sparse training data. The paper proposes a structured approach in which domain-specific samples are augmented and synthesized to create a more comprehensive training set. This augmented data set then feeds into a learning process governed by a Process Reward Model (PRM) that supervises the reasoning process, fostering more stable and reliable outcome generation.

1. Question Augmentation: By reformulating questions and altering option sequences, the available data is expanded, facilitating a wider exploration of the state and action space during the reinforcement learning stage.
2. Reasoning Process Synthesis: To fill in the missing reasoning step data, synthesized steps are generated using stronger reasoning foundation models, allowing the training model to simulate the reasoning steps it will require in future iterations.
3. Few-shot In-Context Learning: Domain-specific samples guide the generation process for domain-specific reasoning tasks, thus embedding these nuances into the model's operational framework.

Empirical Evaluation and Results

OpenRFT's efficacy is validated using the SciKnowEval, a benchmark designed for scientific reasoning tasks spanning multiple disciplines. Notably, OpenRFT exhibits an 11% average performance improvement utilizing only 100 samples per task. This underscores the method's potential for efficient domain adaptation with minimal data. The substantial gain is particularly attributed to the strategic integration of synthesized reasoning data and reinforcement fine-tuning, which enhance the model's adaptability to domain-specific exigencies.

Implications and Future Research Directions

The implications of this paper extend into both practical and theoretical domains. Practically, the methodology facilitates a highly efficient adaptation of large reasoning models to specialized applications, which is crucial for real-world deployment where extensive domain-specific data may be unavailable. Theoretically, the work presents a significant step toward leveraging reasoning foundation models' latent capabilities, pointing to a future where these models possess human-like generalization ability across vastly different problem spaces.

Future work could explore enhancing domain knowledge integration and extending fine-tuning methodologies to handle more complex task requirements. Particularly, advancing PRM functionality to dynamically update and adjust the policy model in response to evolving task conditions could further unlock potential in areas such as autonomous learning environments and interactive AI systems. Additionally, refining data augmentation strategies to intelligently generate context-aware variations holds promise for improving model robustness and generalization further.

In conclusion, OpenRFT represents a methodological advance in adapting reasoning models for domain-specific tasks. It leverages reinforcement learning principles, demonstrating notable improvements over traditional fine-tuning methods and paving the way for more sophisticated and adaptable AI systems.