Reinforcement Fine-Tuning Framework

• RFT is a framework that extends autoregressive models by integrating reinforcement learning with direct reward optimization in sequential, closed-loop settings.
• It employs tailored policy optimization algorithms such as MPO, GRPO, and M-DPO to align model updates with non-differentiable evaluation metrics.
• Empirical studies show that RFT enhances robustness, generalization, and sample efficiency across tasks like multi-agent simulation and vision-language-action integration.

Reinforcement Fine-Tuning (RFT) Framework

Reinforcement Fine-Tuning (RFT) is a framework that adapts next-token prediction models (including large language and behavior models) via direct reward optimization in sequential, closed-loop environments. RFT augments supervised learning protocols by introducing reinforcement-driven policy updates anchored to explicit, often non-differentiable evaluation metrics, subject to distributional regularization. This approach is increasingly adopted across domains such as multi-agent simulation, embodied vision-language-action policies, mathematical reasoning, code generation, and low-level perception. RFT is implemented in variants such as Generalized Relative Policy Optimization (GRPO), Metric-Oriented Policy Optimization (MPO), and Masked Direct Preference Optimization (M-DPO), with reward signals ranging from human labels and verifiable metrics to simulator-defined outcomes. The framework’s key advantage is bridging generalization gaps in out-of-distribution scenarios by aligning models directly to task-grounded rewards, not merely observed data (Pei et al., 28 Sep 2025).

1. Mathematical Formulation and Core Objective

RFT casts autoregressive generation as a stochastic policy in a Markov Decision Process (MDP), where each rollout forms a trajectory $\tau$ scored by a reward $r(\tau)$. The formal loss for RFT (exemplified by SMART-R1 in multi-agent traffic simulation) is defined:

$$L_{\mathrm{MPO}}(\theta) = -\mathbb{E}_{\tau\sim\pi_\theta}[w(\tau)\cdot A(\tau)] + \beta\mathbb{E}[D_{\mathrm{KL}}[\pi_\theta || \pi_\theta^{\mathrm{ref}}]]$$

where:

• $\pi_\theta$ is the current policy,
• $w(\tau) = \pi_\theta(\tau)/\bar{\pi}_\theta(\tau)$ (importance weight with $\bar{\pi}_\theta$ as a no-gradient copy),
• $A(\tau) = r(\tau) - \alpha$ is the one-step advantage with empirical baseline threshold $\alpha$,
• $\beta$ controls KL regularization,
• $$D_{\mathrm{KL}}[\pi_\theta || \pi_\theta^{\mathrm{ref}}]$$ constrains updates to remain close to a reference policy snapshotted before RFT begins (Pei et al., 28 Sep 2025).

This loss encourages higher probability for trajectories with superior reward, while maintaining proximity to the initial supervised distribution. Notably, auxiliary networks (e.g., critics) are omitted, relying instead on metric-driven reward signals and direct policy update.

2. Policy Optimization Algorithms and Training Schedules

RFT is realized through tailored policy optimization algorithms:

• Metric-Oriented Policy Optimization (MPO): Directly optimizes for known, non-differentiable outcome metrics (e.g., Realism Meta in traffic simulation). Rollouts are evaluated in the target environment, advantage is computed relative to baseline, and per-token KL divergence regularizes policy shifts (Pei et al., 28 Sep 2025).
• Generalized Relative Policy Optimization (GRPO): Used broadly in multimodal and reasoning domains, GRPO forms groups of sampled candidate rollouts, normalizes advantages within-group, and applies PPO-style clipping. The policy gradient is computed without value networks (Shi et al., 10 Jun 2025, Liu et al., 3 Mar 2025, Tan et al., 26 Mar 2025).
• Masked Direct Preference Optimization (M-DPO): In mesh generation, face-level masking is used for fine-grained RL, focusing updates on low-quality regions identified by topology-aware metrics (e.g., Boundary Edge Ratio, Topology Score, Hausdorff Distance) (Liu et al., 22 May 2025).
• Adaptive Curriculum Learning (AdaRFT): Dynamic difficulty scheduling is layered on top of RFT algorithms, wherein reward signals adjust the sampling of training tasks to optimize challenge calibration and improve efficiency for reasoning models (Shi et al., 7 Apr 2025).

Training schedules often adopt an iterative SFT–RFT–SFT strategy, in which supervised fine-tuning stabilizes agent rollouts, RFT optimizes realism or task-specific metrics, and a final supervised stage restores fidelity and mitigates catastrophic forgetting (Pei et al., 28 Sep 2025).

3. Reward Design and Evaluation Metrics

Reward signals in RFT frameworks are domain-specific and may be:

• Explicit evaluation metrics: Realism scores, success rates, intersection-over-union for detection, mask GIoU for segmentation, and task-defined meta scores (Pei et al., 28 Sep 2025, Liu et al., 3 Mar 2025, Tan et al., 26 Mar 2025, Zhang et al., 26 Sep 2025).
• Process- and outcome-based rewards: Composite rewards that combine correctness of final output with coherence, rationality, or format compliance of reasoning chains; often leveraging pre-trained verifier models (Zhang et al., 2024, Zhang et al., 19 Feb 2025).
• World model-verified rewards: In simulators, trajectory-level rewards are derived by comparing model rollouts to expert trajectories using perceptual losses within a learned world model (Li et al., 1 Oct 2025).

Policy update loops directly tie probability mass to these rewards, yielding stable and targeted improvements in alignment with safety-critical or domain-specific metrics.

4. Iterative and Unified Training Strategies

Alternation between supervised and reinforcement stages is a recurring motif:

• SFT–RFT–SFT cyclic protocols: Sequentially leverage labeled data to initiate policy, apply RL to advance toward metric alignment, and close with supervised rounds to regain distributional coverage (Pei et al., 28 Sep 2025, Tan et al., 26 Mar 2025).
• Unified Fine-Tuning (UFT): Interleaves supervised hints and RL signals at the trajectory level, breaking the exponential sample complexity bottleneck seen in vanilla RL for long-horizon reasoning. Theoretical analysis shows UFT accelerates convergence to optimal policies for branching and depth-intensive tasks (Liu et al., 22 May 2025).
• Multi-stage RFT: In low-level vision (Refine-IQA), initial stages focus on enhancing perceptual subskills through multi-task rewards, followed by reasoning/interpretation supervision via probability difference rewards that incentivize substantive "think" segments (Jia et al., 4 Aug 2025).

5. Comparison to Standard Supervised Learning and RLHF

Conventional supervised fine-tuning (SFT) minimizes cross-entropy over logged data, which is sample-efficient but fails to optimize for non-differentiable outcome metrics and generalizes poorly under covariate shift or out-of-distribution sampling. RLHF techniques (PPO, DPO) require learned reward models and actor-critic networks, risking instability and sampling bias. RFT directly leverages black-box metrics or verifiable labels, avoids auxiliary critics, applies group-wise KL penalties, and is often embedded in iterative schedules for distributional robustness (Pei et al., 28 Sep 2025, Liu et al., 3 Mar 2025). Empirical evidence shows substantial improvement in realism, reasoning, and data efficiency over both SFT-only and RLHF baselines.

Approach	Reward Source	Auxiliary Networks	Policy Regularization
SFT	Supervised labels	None	None
RLHF (PPO/DPO)	Human preferences	Value/Reward nets	KL to reference or group ranking
RFT (R1-style)	Task metrics	None	Per-token KL to supervised dist.

6. Implementation, Architecture, and Empirical Gains

RFT frameworks are instantiated over diverse neural architectures:

• Next-token predictors: Autoregressive language or motion models (Pei et al., 28 Sep 2025).
• Vision-language-action policies: Vision transformers and LLM-augmented policies with world model simulators (Li et al., 1 Oct 2025).
• Mesh and pose generators: Transformers with hybrid discrete-continuous heads and fine-grained reward masking (Liu et al., 22 May 2025, Li et al., 11 Aug 2025).
• Reasoning and functional token architectures: LLMs enhanced with embedded reasoning tokens for tree-structured exploration (Zhang et al., 19 Feb 2025).

Empirical evaluations consistently demonstrate:

• State-of-the-art realism and alignment on leaderboards (e.g., SMART-R1 with 0.7858 realism meta score, first on WOSAC) (Pei et al., 28 Sep 2025).
• Robust generalization under perturbations and distributional shifts, including sample-efficient adaptation and improved performance in data-scarce or few-shot regimes (Li et al., 1 Oct 2025, Zhang et al., 26 Sep 2025).
• Enhanced reasoning, interpretability, and chain-of-thought quality in both vision and LLMs (Tan et al., 26 Mar 2025, Shi et al., 7 Apr 2025).
• Substantial reduction in sample requirements relative to classic RL pipelines (>10× fewer steps for comparable gains) (Li et al., 1 Oct 2025, Shi et al., 7 Apr 2025).
• Quantitative improvements over baselines, e.g., +13% absolute accuracy increase in geospatial referring, +24.3% accuracy in fine-grained visual classification, superior mesh quality and topology regularity (Zhang et al., 26 Sep 2025, Liu et al., 3 Mar 2025, Liu et al., 22 May 2025).

7. Limitations, Extensions, and Future Directions

Current RFT designs exhibit limitations:

• Reward bias and capacity ceiling: Verified metrics are tied to available expert data and may inhibit extrapolation beyond seen performance (Li et al., 1 Oct 2025).
• Simulator and model fidelity: The quality of learned world models or token encoders constrains ultimate generalization (Li et al., 1 Oct 2025).
• Catastrophic forgetting: Pure RL stages can misalign distributional coverage, addressed via alternating SFT or unified protocols (Pei et al., 28 Sep 2025, Liu et al., 22 May 2025).
• Reward model extensibility: Extending RFT to broader tasks (captioning, dense perception, dialog) may require learned or hybrid reward models (Liu et al., 3 Mar 2025).

Future work may integrate learned critics, richer process-based rewards, dynamic group sizes, improved curriculum learning, and domain-specific augmentation to further enhance alignment, generalization, and sample efficiency. RFT frameworks are rapidly evolving, with active research on deploying these methods for safe AGI, robust real-world simulation, and interpretable reasoning pipelines.