Reinforcement Fine-Tuning (RFT)

• Reinforcement Fine-Tuning (RFT) is a post-training technique that optimizes large models by directly maximizing reward functions.
• It utilizes policy gradient algorithms like PPO and GRPO to encourage exploration, robust generalization, and precise output control.
• RFT enhances applications in reasoning, domain adaptation, and safety while addressing challenges such as vanishing gradients and reward instability.

Reinforcement Fine-Tuning (RFT) is a post-training methodology for large (language or multimodal) models in which parameter updates are guided by maximizing a reward function, typically operationalized via policy gradient algorithms. The key distinction between RFT and classic supervised fine-tuning (SFT) is that, while SFT teaches the model to mimic labeled answers, RFT incentivizes desired behaviors directly through reward signals—potentially enabling richer exploration, robust generalization, and improved alignment for complex reasoning or multi-step decision tasks.

1. Theoretical Foundations and Core Algorithms

RFT is grounded in reinforcement learning (RL) principles. At its heart, RFT formulates model adaptation as policy optimization: for a model parameterized by $\theta$, a policy $\pi_\theta(y|x)$ is optimized to maximize expected reward

$$\max_\theta\, \mathbb{E}_{x\sim\mathcal{D}, y\sim\pi_\theta(y|x)} [ r(x, y, \hat{y}) ]$$

where $r(x, y, \hat{y})$ is a verifiable, usually rule-based reward that often encodes correctness, format, or other desirable criteria (Song et al., 20 May 2025).

Contemporary RFT methods often employ Proximal Policy Optimization (PPO) or related policy gradient algorithms. Many recent variants adopt Group Relative Policy Optimization (GRPO), which estimates the advantage of each sampled output within a group by normalizing its reward:

$$A_i = \frac{r_i - \text{mean}(r_1, \ldots, r_G)}{\text{std}(r_1, \ldots, r_G)}$$

and updates parameters via a clipped objective that also imposes a Kullback-Leibler (KL) divergence penalty to keep the policy close to a reference (e.g., the starting, SFT-initialized model) (Sun et al., 24 May 2025, Luong et al., 17 Jan 2024).

A central and recurring insight is that RFT’s optimization dynamics heavily depend on the reward distribution across model outputs: when the standard deviation of the reward is low (i.e., the rewards are almost equal, even though suboptimal), the gradient can vanish—halting effective learning (Razin et al., 2023).

2. Pipeline Structure and Practical Variants

The canonical RFT pipeline comprises two (or more) stages. First, supervised fine-tuning (SFT) is conducted to give the model basic competence, especially when the reward landscape is poorly shaped. This initial SFT phase is often inexpensive—requiring only a small fraction of labeled data (e.g., 1%) to prevent vanishing gradients and enable reward diversity (Razin et al., 2023).

The second phase is the reinforcement step, where the model’s outputs are sampled on policy, evaluated via reward, and the resulting experience is used to update parameters via policy optimization (PPO, GRPO, or variants). Hybrid pipelines, such as Prefix-RFT, blend SFT and RFT signals during a single training loop, leveraging demonstration prefixes as stable initialization for exploration (Huang et al., 2 Jul 2025).

Algorithmic Structure

Stage	Mechanism	Purpose
SFT	NLL minimization	Provide warm-start, initial diversity
RFT	PPO/GRPO	Optimize for reward, enable exploration
Hybrid (e.g., Prefix-RFT)	Mixed objectives	Combine supervised and RL signals

3. Reward Design and Optimization Obstacles

Reward functions in RFT are typically rule-based, verifiable, and tailored to the domain. For reasoning and multi-step search tasks, they may encompass:

• Correctness of the final answer
• Step-wise or process rewards for intermediate reasoning chains
• Compliance with specified output formats (e.g., presence of reasoning and answer tags)
• Structural properties, such as the validity of a generated DAG in search-augmented QA (Shi et al., 10 Jun 2025)
• Diversity or adversarial signals, especially in red teaming applications (Zheng et al., 4 Jun 2025)

A key theoretical result is that the effectiveness of RFT depends on sufficient variability in rewards across the model's outputs. When this variability is lacking—frequently the case for out-of-distribution inputs or after extensive pretraining—RFT gradients may vanish regardless of how suboptimal the mean reward is, dramatically slowing optimization (Razin et al., 2023). This necessitates either preliminary SFT or explicit reward shaping interventions.

Additionally, outcome-only rewards, while efficient, are sparse and can be unstable; process rewards promise denser feedback but risk introducing instability or reward hacking (Sun et al., 24 May 2025). Several lines of recent research aim to blend outcome and process rewards for robust reasoning and efficient training.

4. Applications Across Domains

RFT has been demonstrated across a range of domains and modalities:

• Mathematical and logical reasoning: Enhances the generalization of LLMs by exposing them to richer reasoning trajectories via sampling and reward-based reinforcement on correctness, outperforming SFT trained on fixed chains-of-thought (Luong et al., 17 Jan 2024).
• Domain adaptation: Methods such as OpenRFT synthesize reasoning-process data, augment questions, and embed few-shot in-context learning to enable adaptation to domain-specific tasks with as few as 100 labeled examples, achieving performance gains of ~11% (Zhang et al., 22 Dec 2024).
• Visual and multi-modal reasoning: Visual-RFT and Reason-RFT show that data-efficient RFT, particularly when coupled with verifiable, task-structured rewards (e.g., IoU for detection tasks), substantially outperforms SFT in object detection, classification, and visual reasoning challenges (Liu et al., 3 Mar 2025, Tan et al., 26 Mar 2025). Novel frameworks extend RFT to video reasoning, mesh generation, and embodied agent planning by exploiting structured outputs and feedback (Wang et al., 18 May 2025, Liu et al., 22 May 2025, Shu et al., 26 May 2025, Tian et al., 26 Jun 2025).
• Continual and stable learning: RFT mitigates catastrophic forgetting when learning novel tasks by steering updates toward high-probability regions of the existing model distribution, whereas SFT on low-likelihood data can induce knowledge erosion (Zhang et al., 30 Jun 2025).
• Safety and red teaming: RFT as a red teaming technique is supported by standardized benchmarks (e.g., RedRFT), with careful intrinsic/extrinsic reward design to balance adversarial generation, diversity, and output safety (Zheng et al., 4 Jun 2025).
• Uncertainty calibration: RFT, absent explicit penalization, can reduce model refusal rates for unanswerable questions—incurring a “hallucination tax”—but can be efficiently corrected by including unanswerable data in the reward function (Song et al., 20 May 2025).

5. Empirical Performance, Efficiency, and Engineering

RFT exhibits both strengths and challenges in empirical performance:

• Generalization and Data Efficiency: RFT pipelines consistently outperform SFT in terms of robustness to data scarcity and out-of-distribution adaptation, requiring fewer labeled samples and excelling in few-shot scenarios (Zhang et al., 22 Dec 2024, Liu et al., 3 Mar 2025).
• Curriculum and Efficiency: Adaptive curriculum learning (AdaRFT) further improves efficiency by selecting training problems in a difficulty band responsive to current performance, accelerating training by up to 2x and improving final accuracy on competition-level math tasks (Shi et al., 7 Apr 2025).
• Stability and Implementation: RFT is sensitive to hyperparameters, initialization, and reward shaping. Implementation frameworks such as Trinity-RFT offer decoupled, modular pipelines supporting synchronous/asynchronous, on-/off-policy, and human-in-the-loop training for research scalability and engineering robustness (Pan et al., 23 May 2025).
• Hybridization: Prefix-RFT and related approaches demonstrate that synergistically blending demonstration-guided prefixes and reward-driven exploration within a unified loop yields improved reliability, robustness, and integration with existing RL and SFT workflows (Huang et al., 2 Jul 2025).
• Prompt Engineering: Prior prompt engineering during RFT (pPE) can steer internalized model behavior more effectively than inference-time prompt modifications, and distinct behavioral tags in the training prompt yield measurable differences in reasoning, planning, and coding styles after RFT (Taveekitworachai et al., 20 May 2025, Cen et al., 25 May 2025).

6. Known Limitations and Open Challenges

Despite its demonstrated potential, RFT faces several well-documented limitations:

• Vanishing Gradients: When the reward distribution over model outputs is narrow, RFT stalls even when the expected reward is suboptimal, necessitating preemptive SFT or deliberate reward shaping (Razin et al., 2023).
• Behavioral Instability: RFT may induce unexpected or unsafe behaviors due to reward hacking, overfitting to process rewards, or degradation in refusal capabilities (“hallucination tax”) (Song et al., 20 May 2025).
• Sensitivity to Data and Initialization: The benefit from RFT is highly dependent on data distribution, reward signal quality, and the base model’s “RL readiness,” including the informativeness and diversity of SFT or demonstration data (Cen et al., 25 May 2025, Zhang et al., 30 Jun 2025).
• Scalability & System Demands: Efficient implementation of RFT at scale requires sophisticated system support (experience replay buffers, decoupled rollouts, robust data handling), and managing training stability under asynchronous RL workloads (Pan et al., 23 May 2025).

7. Future Directions

Several research avenues are identified as central for advancing RFT:

• Unified and Hybrid Paradigms: Further harmonizing demonstration-driven and exploration-driven fine-tuning—potentially via dynamic or curriculum-based prefix sampling, or adaptive mixing of SFT/RFT signals (Huang et al., 2 Jul 2025, Shi et al., 7 Apr 2025).
• Process vs. Outcome Reward Blending: Developing theoretical and empirical foundations for integrating dense process rewards with sparse outcome rewards to ensure both robust exploration and stable final performance (Sun et al., 24 May 2025).
• Automated and Data-efficient Reasoning: Improving RFT’s data efficiency via synthesized reasoning-process generation, in-context knowledge embedding, and curriculum tuning for low-resource scenarios (Zhang et al., 22 Dec 2024, Liu et al., 3 Mar 2025).
• Safety, Uncertainty, and Alignment: Ensuring RFT-trained models maintain epistemic humility and safety, with rigorous benchmarks and reward regularization to penalize overconfident hallucinations or adversarial outputs (Song et al., 20 May 2025, Zheng et al., 4 Jun 2025).
• Continual Learning: Leveraging RFT’s tendency to reinforce high-probability rollouts for stable adaptation to novel tasks while minimizing catastrophic forgetting, and systematically studying the role of data distribution in lifelong learning (Zhang et al., 30 Jun 2025).
• Multimodal and Embodied Reasoning: Extending RFT frameworks to encompass video, 3D, and embodied environments, exploiting advances in GRPO, tree-based optimization, and reward model generalization for self-evolving agents (Shu et al., 26 May 2025, Tian et al., 26 Jun 2025).

Reinforcement Fine-Tuning has emerged as a foundational paradigm for aligning models with verifiable objectives across diverse modalities and domains. Its success hinges on careful reward function design, data distribution management, and algorithmic strategies that combine robust exploration, efficient generalization, and stable knowledge retention.