RFT Selection: Reinforcement Fine-Tuning Rationale
- Reason-to-Select RFT is a post-training strategy that optimizes task-specific rewards over token imitation, addressing objective mismatch in downstream evaluations.
- It facilitates the exploration of diverse reasoning trajectories, overcoming the limitations of chain-of-thought supervision to boost adaptability.
- RFT employs a two-stage pipeline with a supervised warm start followed by reward-driven optimization, yielding empirical gains in areas like math and visual reasoning.
“Reason-to-Select RFT” most commonly refers, in the current arXiv literature, to the conditions under which reinforcement fine-tuning is preferred to purely supervised post-training for LLMs and MLLMs: chiefly when downstream success is better expressed by a reward than by exact token imitation, when multiple valid reasoning trajectories exist, and when exploration, self-correction, or action-consequence optimization are central to the task (Luong et al., 2024, Tan et al., 26 Mar 2025, Sun et al., 24 May 2025). The acronym is not uniform across all fields: one data-selection paper uses RFT for rejection fine-tuning (Li et al., 21 May 2026), and a lattice-QCD paper uses RFT for the relativistic-field-theoretic finite-volume formalism (Alotaibi et al., 2024). In the post-2024 reasoning literature, however, the dominant sense is reinforcement fine-tuning.
1. Objective mismatch as the primary selection criterion
The most recurrent reason to select reinforcement fine-tuning is that token-level cross-entropy is often misaligned with downstream evaluation. This contrast is stated explicitly in work on video-based 3D scene understanding, where supervised fine-tuning is written as
while reinforcement fine-tuning is framed as direct reward optimization,
The paper’s argument is that textual next-token imitation is only an indirect proxy when evaluation is ultimately performed in continuous or structured spaces such as 3D IoU, F1-score, frame localization, or exact answer accuracy after decoding (Linghu et al., 5 Mar 2026).
This alignment argument recurs across domains. In screenshot-based UI grounding, any predicted point inside the target box is correct, but SFT still penalizes all outputs except the reference point, so the paper selects GRPO-based RFT to optimize the task criterion directly rather than the textual serialization of one target coordinate (Gu et al., 14 Aug 2025). In visual perception tasks, Visual-RFT makes the same move by replacing answer imitation with verifiable rewards such as exact class match, IoU reward, confidence reward, and format reward (Liu et al., 3 Mar 2025). In VLA control, VLA-RFT argues that imitation learning optimizes action matching on dataset states, whereas RFT optimizes action sequences by their rollout consequences in a world model with dense, trajectory-level verified rewards (Li et al., 1 Oct 2025).
Taken together, these works suggest that RFT is selected when evaluation is geometric, executable, trajectory-level, or otherwise many-to-one with respect to textual outputs. In such settings, cross-entropy can punish task-correct outputs, while reward optimization can target the metric that actually determines success.
2. Exploration, multiplicity of reasoning paths, and the limits of static CoT supervision
A second major reason to select RFT is that reasoning tasks rarely admit only one useful chain of thought. ReFT formulates this point most directly: conventional CoT-SFT learns from one annotated reasoning path per question, whereas reinforcement fine-tuning can sample an “abundance of reasoning paths” from the same training questions and reinforce those that end in the correct answer (Luong et al., 2024). The paper treats this as a generalization argument rather than a pure data-scaling argument: improvement is obtained from the same training questions, without extra or augmented questions.
Reason-RFT makes the analogous argument for visual reasoning. Its diagnosis of CoT-SFT is “overfitting,” “cognitive rigidity,” and weak transfer from in-domain to out-of-domain tasks. The proposed two-phase recipe uses a small CoT activation stage only to place the policy in a reasoning-capable regime, then GRPO to sample multiple reasoning-response pairs and optimize against rule-based rewards, thereby encouraging more adaptive visual reasoning behavior (Tan et al., 26 Mar 2025). Point-RFT applies the same logic to visual document reasoning, but adds a multimodal critique: text-only CoT induces “visual hallucinations and insufficient multimodal integration,” so the selected RFT regime is one that explores grounded CoT trajectories with explicit point references to visual evidence (Ni et al., 26 May 2025).
Several specialized variants sharpen this selection logic. RFTT argues that direct RL over the full token vocabulary makes reasoning search combinatorial and inefficient, so it replaces unconstrained token-level exploration with a small learned action space of functional tokens such as <analysis>, <verify>, and <refine> (Zhang et al., 19 Feb 2025). RA-RFT argues that standard retrieval is poorly matched to reasoning because semantic similarity and strategy similarity diverge; it therefore selects retrieval-augmented RFT so the policy can learn, under outcome rewards, how to exploit retrieved analogous demonstrations rather than merely seeing them in context (Xiao et al., 11 Jun 2026). Mesh-RFT makes a parallel point for 3D mesh generation: global object-level preferences are too coarse because low-quality and high-quality regions coexist within the same mesh, so it selects a fine-grained preference-optimization variant that localizes learning to face-level defects (Liu et al., 22 May 2025).
The unifying theme is that RFT is selected when the search space of correct solutions is broader than the annotation space, and when useful learning depends on exploring alternatives rather than copying one fixed demonstration.
3. Canonical pipeline and major algorithmic families
The dominant recipe in this literature is a two-stage pipeline: a supervised warm start followed by reward-driven optimization. This appears in ReFT, RFTT, Reason-RFT, Point-RFT, 3D-RFT, VideoRFT, RM-R1, and many related systems (Luong et al., 2024, Zhang et al., 19 Feb 2025, Tan et al., 26 Mar 2025, Wang et al., 18 May 2025, Chen et al., 5 May 2025, Linghu et al., 5 Mar 2026). The warm start teaches output format, activates task-specific reasoning behavior, or initializes a stable policy; the reinforcement stage then searches over alternative trajectories and optimizes them against task-aligned rewards.
Within that general template, GRPO has become the most visible critic-free optimizer. In its common form, group-normalized advantage is
so each sampled trajectory is judged relative to its peers from the same prompt or state (Tan et al., 26 Mar 2025). The appeal of GRPO in these papers is practical as much as conceptual: it avoids a separate value network, reduces memory, and fits rule-based or verifier-based whole-trajectory rewards.
The family of methods grouped under “RFT” is broader than a single optimizer. Some papers use PPO-style policy gradients with KL regularization to an SFT reference model (Luong et al., 2024, Zhang et al., 19 Feb 2025). Some emphasize RLVR, where reward is computed by deterministic verification rather than a learned reward model (Chen et al., 5 May 2025, Linghu et al., 5 Mar 2026). Some use structured preference optimization rather than online RL, as in Mesh-RFT’s Masked DPO (Liu et al., 22 May 2025). Others extend RFT with explicit reasoning interfaces: functional tokens in RFTT, grounded point traces in Point-RFT, analogous retrieval in RA-RFT, semantic-consistency reward in VideoRFT, or verified world-model rollouts in VLA-RFT (Zhang et al., 19 Feb 2025, Ni et al., 26 May 2025, Xiao et al., 11 Jun 2026, Wang et al., 18 May 2025, Li et al., 1 Oct 2025).
This diversity indicates that “reason-to-select RFT” is not merely a choice of optimizer. It is a choice of post-training principle: optimize reasoning or action policies against success criteria that are closer to deployment metrics than supervised imitation alone.
4. Empirical evidence across domains
The empirical case for selecting RFT is strongest in structured reasoning domains, especially when data are scarce, OOD transfer matters, or evaluation is verifiable.
| Setting | Reported change | Source |
|---|---|---|
| Mathematical reasoning on MATH | Qwen-2.5-7B-Instruct: 70.6% → 79.8%; LLaMA-3.1-8B-Instruct: 32.2% → 60.2% | (Zhang et al., 19 Feb 2025) |
| Visual reasoning OOD | Super-CLEVR: 39.2 → 51.2 over ANS-SFT; Geometry3K: 25.25 → 33.13 over CoT-SFT | (Tan et al., 26 Mar 2025) |
| Visual document QA | ChartQA: 70.88% → 90.04%; text-only CoT RFT reaches 83.92% | (Ni et al., 26 May 2025) |
| Video-based 3D scene understanding | ScanNetDetection : 38.2 → 43.7; ScanRefer [email protected]: 36.4 → 42.9 | (Linghu et al., 5 Mar 2026) |
| Low-shot visual perception | One-shot fine-grained classification: +24.3 over baseline; COCO two-shot: +21.9; LVIS: +15.4 | (Liu et al., 3 Mar 2025) |
| VLA control | LIBERO average success rate: 86.6% → 91.1% with 400 fine-tuning steps | (Li et al., 1 Oct 2025) |
| UI navigation | AndroidWorld success rate: 49.1% for 7B and 65.9% for 72B | (Gu et al., 14 Aug 2025) |
| Retrieval-augmented math reasoning | AIME 2025 average@32 improves over GRPO by 7.1 points for Qwen3-1.7B and 2.8 for Qwen3-4B | (Xiao et al., 11 Jun 2026) |
Beyond these headline results, low-resource domain adaptation provides a further selection argument. OpenRFT reports average accuracy improving from 0.403 to 0.447 with only 100 training samples per task on SciKnowEval, after combining SFT initialization, PRM-guided PPO, and domain-data reuse via augmentation and process synthesis (Zhang et al., 2024). At the data-selection level, HES-based RFT shows that not all correct trajectories are equally useful: Highest-HES selection outperforms random, while Lowest-HES global-pool selection can fall to 10.85–15.59 average, indicating that poor successful trajectories can actively contaminate rejection-style fine-tuning (Li et al., 21 May 2026).
The broad empirical pattern is consistent. The gains are usually largest when tasks are verifiable, reasoning-heavy, low-resource, or OOD-shifted, and they are often especially pronounced for small or medium open models rather than only frontier-scale closed systems.
5. Data curation, systems infrastructure, and the operational case for RFT
A practical reason to select RFT is that the surrounding infrastructure has become mature enough to make it operational rather than merely conceptual. Trinity-RFT presents this most explicitly as a general-purpose framework with a decoupled explorer–buffer–trainer architecture supporting synchronous and asynchronous, on-policy and off-policy, and online and offline regimes in one system (Pan et al., 23 May 2025). The paper’s claim is infrastructural: real RFT workloads mix rollout collection, delayed rewards, offline corpora, human feedback, and distributed training, so a usable RFT stack must support more than a monolithic PPO loop.
RedRFT makes a narrower but complementary point for red teaming. It treats adversarial prompt generation as a sequential decision problem and argues that PPO-based RFT is highly sensitive to implementation details such as rollout batch size, LoRA, KL regularization, and Lagrange-multiplier updates for constraints (Zheng et al., 4 Jun 2025). The paper’s contribution is a benchmark and modular implementation rather than a new theoretical reason to prefer RL, but it strengthens the broader selection case by showing that RFT can be standardized, ablated, and rapidly prototyped.
Operational papers also show that data quality and data selection are not peripheral. UI-Venus reports that roughly 40% of current open-source grounding data contained significant noise and explicitly states that RFT training requires high-quality clean data, leading it to retain about 107k grounding samples from a much larger collected pool (Gu et al., 14 Aug 2025). Point-RFT relies on a curated 71K grounded-CoT corpus before RL (Ni et al., 26 May 2025). OpenRFT relies on question augmentation, synthesized reasoning traces, and PRM-guided PPO because low-resource domain adaptation is otherwise too sparse (Zhang et al., 2024). HES generalizes that idea by arguing that RFT depends not just on generating correct candidates, but on selecting which successful trajectories deserve to become supervision (Li et al., 21 May 2026).
This suggests that selecting RFT also means selecting an accompanying regime of reward engineering, data filtering, rollout infrastructure, and trajectory curation. In the surveyed literature, the strongest RFT results are rarely obtained by reward optimization alone.
6. Limits, caveats, and decision boundaries
The surveyed papers are broadly favorable to RFT, but they also define the conditions under which it is less attractive. The first limitation is complexity. RFTT requires a two-phase pipeline, vocabulary expansion with learnable functional tokens, prompt-guided tree search for warmup data, and online RL with MCTS (Zhang et al., 19 Feb 2025). Reason-RFT requires CoT activation, task-specific reward design, and GRPO training, and it documents reward sensitivity, transient adaptation gaps in RL-only variants, and reasoning redundancy at inference (Tan et al., 26 Mar 2025). Point-RFT notes substantial compute cost, added latency from sequential grounded reasoning, and dependence on a high-quality grounded rationale dataset (Ni et al., 26 May 2025).
The second limitation is reward quality. ReFT shows a clear reward-hacking failure mode on multiple-choice MathQA, where final-answer reward can reinforce flawed reasoning if the answer space is small (Luong et al., 2024). OpenRFT warns that final correctness alone can reinforce faulty intermediate reasoning, which is why it adds PRM-based process supervision and highlights teacher–student action-space alignment as a practical constraint (Zhang et al., 2024). RM-R1 likewise argues that cold-start RL alone is insufficient for generalist reward modeling and requires high-quality reasoning-trace distillation before RLVR can generalize well (Chen et al., 5 May 2025).
The third limitation is domain scope. Several papers explicitly acknowledge that evidence remains concentrated in structured reasoning domains. RFTT’s strongest validation is on mathematical reasoning, and the authors state that broader reasoning domains remain to be studied (Zhang et al., 19 Feb 2025). 3D-RFT argues strongly for metrics-driven policy optimization in video-based 3D scene understanding, but it is equally explicit that its reward design relies on strictly verifiable geometric metrics such as 3D IoU and F1-score (Linghu et al., 5 Mar 2026). VLA-RFT depends on world-model fidelity and on rewards defined by similarity to goal-achieving reference trajectories, which constrains how far the learned policy can depart from expert-style behavior (Li et al., 1 Oct 2025).
A plausible implication is that RFT is least attractive when simplicity matters more than peak performance, when reliable verifiable rewards are unavailable, when reward hacking is hard to control, or when the target domain is far from the structured settings in which these gains were demonstrated. Under those conditions, SFT, DPO, or narrower post-training methods may remain preferable. Where rewards are reliable, trajectories are meaningfully searchable, and deployment metrics diverge from exact sequence imitation, however, the recent literature consistently treats RFT as the more appropriate selection.