---
title: 'RFT Selection: Reinforcement Fine-Tuning Rationale'
url: https://www.emergentmind.com/topics/reason-to-select-rft
type: topic
---

# RFT Selection: Reinforcement Fine-Tuning Rationale

“Reason-to-Select RFT” most commonly refers, in the current arXiv literature, to the conditions under which **reinforcement fine-tuning** is preferred to purely supervised post-training for LLMs and MLLMs: chiefly when downstream success is better expressed by a reward than by exact token imitation, when multiple valid reasoning trajectories exist, and when exploration, self-correction, or action-consequence optimization are central to the task [2401.08967; 2503.20752; 2505.18536]. The acronym is not uniform across all fields: one data-selection paper uses **RFT** for **rejection fine-tuning** [2605.22389], and a lattice-QCD paper uses **RFT** for the **relativistic-field-theoretic** finite-volume formalism [2412.05060]. In the post-2024 reasoning literature, however, the dominant sense is reinforcement fine-tuning.

## 1. Objective mismatch as the primary selection criterion

The most recurrent reason to select reinforcement fine-tuning is that **token-level cross-entropy is often misaligned with downstream evaluation**. This contrast is stated explicitly in work on video-based 3D scene understanding, where supervised fine-tuning is written as
$$
\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{t=1}^{T} \log \pi_\theta(\mathbf{y^*_t} \mid \mathbf{x}, \mathbf{I}, \mathbf{y^*_{<t}})
$$
while reinforcement fine-tuning is framed as direct reward optimization,
$$
J(\theta) = \mathbb{E}_{\mathbf{x}\sim\mathcal{D},\mathbf{y}\sim \pi_{\theta}(\cdot |\mathbf{x})}[R].
$$
The paper’s argument is that textual next-token imitation is only an indirect proxy when evaluation is ultimately performed in continuous or structured spaces such as 3D IoU, F1-score, frame localization, or exact answer accuracy after decoding [2603.04976].

This alignment argument recurs across domains. In screenshot-based UI grounding, any predicted point inside the target box is correct, but SFT still penalizes all outputs except the reference point, so the paper selects GRPO-based RFT to optimize the task criterion directly rather than the textual serialization of one target coordinate [2508.10833]. In visual perception tasks, Visual-RFT makes the same move by replacing answer imitation with verifiable rewards such as exact class match, IoU reward, confidence reward, and format reward [2503.01785]. In VLA control, VLA-RFT argues that imitation learning optimizes action matching on dataset states, whereas RFT optimizes action sequences by their rollout consequences in a world model with dense, trajectory-level verified rewards [2510.00406].

Taken together, these works suggest that RFT is selected when evaluation is **geometric, executable, trajectory-level, or otherwise many-to-one with respect to textual outputs**. In such settings, cross-entropy can punish task-correct outputs, while reward optimization can target the metric that actually determines success.

## 2. Exploration, multiplicity of reasoning paths, and the limits of static CoT supervision

A second major reason to select RFT is that reasoning tasks rarely admit only one useful chain of thought. ReFT formulates this point most directly: conventional CoT-SFT learns from one annotated reasoning path per question, whereas reinforcement fine-tuning can sample an “abundance of reasoning paths” from the same training questions and reinforce those that end in the correct answer [2401.08967]. The paper treats this as a generalization argument rather than a pure data-scaling argument: improvement is obtained from the same training questions, without extra or augmented questions.

Reason-RFT makes the analogous argument for visual reasoning. Its diagnosis of CoT-SFT is “overfitting,” “cognitive rigidity,” and weak transfer from in-domain to out-of-domain tasks. The proposed two-phase recipe uses a small CoT activation stage only to place the policy in a reasoning-capable regime, then GRPO to sample multiple reasoning-response pairs and optimize against rule-based rewards, thereby encouraging more adaptive visual reasoning behavior [2503.20752]. Point-RFT applies the same logic to visual document reasoning, but adds a multimodal critique: text-only CoT induces “visual hallucinations and insufficient multimodal integration,” so the selected RFT regime is one that explores grounded CoT trajectories with explicit point references to visual evidence [2505.19702].

Several specialized variants sharpen this selection logic. RFTT argues that direct RL over the full token vocabulary makes reasoning search combinatorial and inefficient, so it replaces unconstrained token-level exploration with a small learned action space of functional tokens such as `<analysis>`, `<verify>`, and `<refine>` [2502.13389]. RA-RFT argues that standard retrieval is poorly matched to reasoning because semantic similarity and strategy similarity diverge; it therefore selects retrieval-augmented RFT so the policy can learn, under outcome rewards, how to exploit retrieved analogous demonstrations rather than merely seeing them in context [2606.13680]. Mesh-RFT makes a parallel point for 3D mesh generation: global object-level preferences are too coarse because low-quality and high-quality regions coexist within the same mesh, so it selects a fine-grained preference-optimization variant that localizes learning to face-level defects [2505.16761].

The unifying theme is that RFT is selected when **the search space of correct solutions is broader than the annotation space**, and when useful learning depends on exploring alternatives rather than copying one fixed demonstration.

## 3. Canonical pipeline and major algorithmic families

The dominant recipe in this literature is a **two-stage pipeline**: a supervised warm start followed by reward-driven optimization. This appears in ReFT, RFTT, Reason-RFT, Point-RFT, 3D-RFT, VideoRFT, RM-R1, and many related systems [2401.08967; 2502.13389; 2503.20752; 2505.12434; 2505.02387; 2603.04976]. The warm start teaches output format, activates task-specific reasoning behavior, or initializes a stable policy; the reinforcement stage then searches over alternative trajectories and optimizes them against task-aligned rewards.

Within that general template, **GRPO** has become the most visible critic-free optimizer. In its common form, group-normalized advantage is
$$
A_i = \frac{R_i - \mathrm{mean}(\{R_1,\dots,R_G\})}{\mathrm{std}(\{R_1,\dots,R_G\})},
$$
so each sampled trajectory is judged relative to its peers from the same prompt or state [2503.20752]. The appeal of GRPO in these papers is practical as much as conceptual: it avoids a separate value network, reduces memory, and fits rule-based or verifier-based whole-trajectory rewards.

The family of methods grouped under “RFT” is broader than a single optimizer. Some papers use PPO-style policy gradients with KL regularization to an SFT reference model [2401.08967; 2502.13389]. Some emphasize RLVR, where reward is computed by deterministic verification rather than a learned reward model [2505.02387; 2603.04976]. Some use structured preference optimization rather than online RL, as in Mesh-RFT’s Masked DPO [2505.16761]. Others extend RFT with explicit reasoning interfaces: functional tokens in RFTT, grounded point traces in Point-RFT, analogous retrieval in RA-RFT, semantic-consistency reward in VideoRFT, or verified world-model rollouts in VLA-RFT [2502.13389; 2505.19702; 2606.13680; 2505.12434; 2510.00406].

This diversity indicates that “reason-to-select RFT” is not merely a choice of optimizer. It is a choice of **post-training principle**: optimize reasoning or action policies against success criteria that are closer to deployment metrics than supervised imitation alone.

## 4. Empirical evidence across domains

The empirical case for selecting RFT is strongest in structured reasoning domains, especially when data are scarce, OOD transfer matters, or evaluation is verifiable.

| Setting | Reported change | Source |
|---|---|---|
| Mathematical reasoning on MATH | Qwen-2.5-7B-Instruct: **70.6% → 79.8%**; LLaMA-3.1-8B-Instruct: **32.2% → 60.2%** | [2502.13389] |
| Visual reasoning OOD | Super-CLEVR: **39.2 → 51.2** over ANS-SFT; Geometry3K: **25.25 → 33.13** over CoT-SFT | [2503.20752] |
| Visual document QA | ChartQA: **70.88% → 90.04%**; text-only CoT RFT reaches **83.92%** | [2505.19702] |
| Video-based 3D scene understanding | ScanNetDetection \(F1_{25}\): **38.2 → 43.7**; ScanRefer Acc@0.25: **36.4 → 42.9** | [2603.04976] |
| Low-shot visual perception | One-shot fine-grained classification: **+24.3** over baseline; COCO two-shot: **+21.9**; LVIS: **+15.4** | [2503.01785] |
| VLA control | LIBERO average success rate: **86.6% → 91.1%** with **400** fine-tuning steps | [2510.00406] |
| UI navigation | AndroidWorld success rate: **49.1%** for 7B and **65.9%** for 72B | [2508.10833] |
| Retrieval-augmented math reasoning | AIME 2025 average@32 improves over GRPO by **7.1** points for Qwen3-1.7B and **2.8** for Qwen3-4B | [2606.13680] |

Beyond these headline results, low-resource domain adaptation provides a further selection argument. OpenRFT reports average accuracy improving from **0.403** to **0.447** with only **100** training samples per task on SciKnowEval, after combining SFT initialization, PRM-guided PPO, and domain-data reuse via augmentation and process synthesis [2412.16849]. At the data-selection level, HES-based RFT shows that not all correct trajectories are equally useful: Highest-HES selection outperforms random, while Lowest-HES global-pool selection can fall to **10.85–15.59** average, indicating that poor successful trajectories can actively contaminate rejection-style fine-tuning [2605.22389].

The broad empirical pattern is consistent. The gains are usually largest when tasks are **verifiable, reasoning-heavy, low-resource, or OOD-shifted**, and they are often especially pronounced for small or medium open models rather than only frontier-scale closed systems.

## 5. Data curation, systems infrastructure, and the operational case for RFT

A practical reason to select RFT is that the surrounding infrastructure has become mature enough to make it operational rather than merely conceptual. Trinity-RFT presents this most explicitly as a general-purpose framework with a decoupled **explorer–buffer–trainer** architecture supporting synchronous and asynchronous, on-policy and off-policy, and online and offline regimes in one system [2505.17826]. The paper’s claim is infrastructural: real RFT workloads mix rollout collection, delayed rewards, offline corpora, human feedback, and distributed training, so a usable RFT stack must support more than a monolithic PPO loop.

RedRFT makes a narrower but complementary point for red teaming. It treats adversarial prompt generation as a sequential decision problem and argues that PPO-based RFT is highly sensitive to implementation details such as rollout batch size, LoRA, KL regularization, and Lagrange-multiplier updates for constraints [2506.04302]. The paper’s contribution is a benchmark and modular implementation rather than a new theoretical reason to prefer RL, but it strengthens the broader selection case by showing that RFT can be standardized, ablated, and rapidly prototyped.

Operational papers also show that **data quality and data selection are not peripheral**. UI-Venus reports that roughly **40%** of current open-source grounding data contained significant noise and explicitly states that RFT training requires high-quality clean data, leading it to retain about **107k** grounding samples from a much larger collected pool [2508.10833]. Point-RFT relies on a curated **71K** grounded-CoT corpus before RL [2505.19702]. OpenRFT relies on question augmentation, synthesized reasoning traces, and PRM-guided PPO because low-resource domain adaptation is otherwise too sparse [2412.16849]. HES generalizes that idea by arguing that RFT depends not just on generating correct candidates, but on selecting which successful trajectories deserve to become supervision [2605.22389].

This suggests that selecting RFT also means selecting an accompanying regime of **reward engineering, data filtering, rollout infrastructure, and trajectory curation**. In the surveyed literature, the strongest RFT results are rarely obtained by reward optimization alone.

## 6. Limits, caveats, and decision boundaries

The surveyed papers are broadly favorable to RFT, but they also define the conditions under which it is less attractive. The first limitation is **complexity**. RFTT requires a two-phase pipeline, vocabulary expansion with learnable functional tokens, prompt-guided tree search for warmup data, and online RL with MCTS [2502.13389]. Reason-RFT requires CoT activation, task-specific reward design, and GRPO training, and it documents reward sensitivity, transient adaptation gaps in RL-only variants, and reasoning redundancy at inference [2503.20752]. Point-RFT notes substantial compute cost, added latency from sequential grounded reasoning, and dependence on a high-quality grounded rationale dataset [2505.19702].

The second limitation is **reward quality**. ReFT shows a clear reward-hacking failure mode on multiple-choice MathQA, where final-answer reward can reinforce flawed reasoning if the answer space is small [2401.08967]. OpenRFT warns that final correctness alone can reinforce faulty intermediate reasoning, which is why it adds PRM-based process supervision and highlights teacher–student action-space alignment as a practical constraint [2412.16849]. RM-R1 likewise argues that cold-start RL alone is insufficient for generalist reward modeling and requires high-quality reasoning-trace distillation before RLVR can generalize well [2505.02387].

The third limitation is **domain scope**. Several papers explicitly acknowledge that evidence remains concentrated in structured reasoning domains. RFTT’s strongest validation is on mathematical reasoning, and the authors state that broader reasoning domains remain to be studied [2502.13389]. 3D-RFT argues strongly for metrics-driven policy optimization in video-based 3D scene understanding, but it is equally explicit that its reward design relies on strictly verifiable geometric metrics such as 3D IoU and F1-score [2603.04976]. VLA-RFT depends on world-model fidelity and on rewards defined by similarity to goal-achieving reference trajectories, which constrains how far the learned policy can depart from expert-style behavior [2510.00406].

A plausible implication is that RFT is least attractive when **simplicity matters more than peak performance**, when **reliable verifiable rewards are unavailable**, when **reward hacking is hard to control**, or when the target domain is far from the structured settings in which these gains were demonstrated. Under those conditions, SFT, DPO, or narrower post-training methods may remain preferable. Where rewards are reliable, trajectories are meaningfully searchable, and deployment metrics diverge from exact sequence imitation, however, the recent literature consistently treats RFT as the more appropriate selection.

Source: https://www.emergentmind.com/topics/reason-to-select-rft