---
title: Reward-Ranked Fine-Tuning (RAFT)
url: https://www.emergentmind.com/topics/reward-ranked-fine-tuning-raft
type: topic
---

# Reward-Ranked Fine-Tuning (RAFT)

Reward-Ranked Fine-Tuning (RAFT) is a family of methods for aligning generative models—particularly large language models (LLMs) and diffusion models—with externally specified preferences or reward functions. RAFT algorithms select and rank candidate outputs according to a reward model (often trained from human feedback or preference data) and subsequently fine-tune the model on the most highly rewarded outputs. This approach aims to achieve high alignment and sample efficiency while remaining computationally simpler and more stable than traditionally on-policy reinforcement learning from human feedback (RLHF) algorithms such as PPO.

## 1. Fundamental Principles of RAFT

RAFT’s central principle is to decouple sample generation from policy (model) update, in contrast to standard RLHF methods which rely on on-policy gradients and tightly coupled actor–critic training. The canonical RAFT workflow involves, for each input prompt:
- Generating $K$ candidate model outputs,
- Ranking or scoring them with a reward model,
- Selecting the highest-reward samples,
- Fine-tuning the model with supervised objectives on the selected subset.

This “best-of-K” or rejection sampling strategy forms the core of RAFT [2304.06767, 2504.11343]. Rather than updating on both positive and negative samples (or on reward differences as in PPO), RAFT fine-tunes the model primarily on positively rewarded outcomes. The reward model itself is typically trained via pairwise preference comparisons, e.g., with a Bradley–Terry objective,
\[
p^*\big(y_w > y_l \mid x\big) = \frac{\exp r^*(x, y_w)}{\exp r^*(x, y_w) + \exp r^*(x, y_l)}
\]
where $r^*(x, y)$ is the learned reward for $(x, y)$.

## 2. RAFT Algorithmic Frameworks and Variants

### 2.1 Canonical Algorithm (Language Models)

The standard RAFT procedure for LLMs is:
1. **Sample Generation:** For each prompt $x$, sample $K$ candidate completions $\{y_1, ..., y_K\}$ from the current model.
2. **Scoring and Filtering:** Evaluate each $y_k$ with a reward model $r(x, y_k)$. Keep only those with the highest rewards (e.g., $r=1$).
3. **Supervised Update:** Fine-tune the model with maximum likelihood or log-likelihood loss on the filtered (positive) examples:
   \[
   \mathcal{L}^{\mathrm{RAFT}}(\theta) = \sum_{(x, a)\in \mathcal{D}} \log \pi_\theta(a|x)
   \]
   where $\mathcal{D}$ is the set of accepted pairs.
4. **Iteration:** Repeat the above with the updated model.

This approach eliminates negative rewards during optimization, focusing purely on high-quality responses [2504.11343]. Empirically, this holds comparable performance to more complex RL approaches such as GRPO and PPO in early stages, though long-term policy entropy may decrease due to lack of negative samples.

### 2.2 RAFT for Diffusion Models

For generative diffusion models, RAFT-inspired fine-tuning can be adapted using different mechanisms depending on reward differentiability:
- **Ranking Loss or Reward Filtering:** For non-differentiable rewards, generated samples are filtered and the model is fine-tuned on those with the highest reward (analogous to rejection sampling).
- **DRaFT Integration:** For differentiable rewards, one can directly backpropagate the reward gradient through the (fully or partially unrolled) sampling chain (cf. equations (1), (2), and variants DRaFT-K and DRaFT-LV) [2309.17400]. Hybrid losses combining direct reward optimization and ranking loss can be used:
  \[
  L_{\text{total}} = \lambda \cdot \big(-r(\text{sample}(\theta, c, x_T), c)\big) + (1-\lambda) \cdot L_{\text{rank}}
  \]
  where $L_{\text{rank}}$ is a ranking loss over generated samples.

### 2.3 Curriculum and Sample Efficiency Extensions

Recent work augments RAFT with prompt-specific inference budgeting to accelerate convergence (GVM-RAFT) [2505.02391]. Sample budgets $n_i^t$ per prompt are dynamically allocated according to both acceptance rate and the gradient norm, minimizing variance:
\[
n_i^t = N \cdot \frac{G_i / \sqrt{p_i^t + \alpha / (p_i^t)^{\beta-1}}}{\sum_{l=1}^m G_l / \sqrt{p_l^t + \alpha / (p_l^t)^{\beta-1}}}
\]
where $p_i^t$ is the acceptance rate, $G_i$ is the gradient norm, and $N$ is the total sample budget.

## 3. Reward Modeling and Ranking Approaches

RAFT’s efficacy hinges on the quality of its reward model, which supplies the ranking or scoring signals. There are several technical variants:
- **Pairwise Preference Models:** Trained on human-labeled (chosen, rejected) response pairs, using Bradley–Terry or similar objectives.
- **Proto-RM (Prototypical Reward Models):** Utilize prototypical networks to enable robust reward estimation from small data, grouping examples by prototype vectors and leveraging weighted prototype averages for reward estimation [2406.06606].
- **TinyRM:** Employs small, encoder-only masked language models (MLMs) as lightweight reward models with FLAN-style prompting and Directional LoRA adaptation. These models demonstrate strong performance in reasoning and safety, providing efficient and scalable ranking signals for RAFT with substantial resource reduction [2507.09973].
  
The practical reward loss typically takes the form:
\[
\mathcal{L}_r = -\mathbb{E}_{(x_i, y_i^+, y_i^-)}\left[ \log \sigma\big(r_\phi(x_i, y_i^+) - r_\phi(x_i, y_i^-)\big)\right]
\]
where $r_\phi$ is the reward model and $\sigma$ the sigmoid function.

## 4. Applications and Implications

RAFT-type fine-tuning is applied across several domains:
- **LLM Alignment:** RAFT aligns LLM outputs with human preferences, practical ethics, or task-specific objectives, while maintaining sample efficiency and simplifying implementation relative to RLHF [2304.06767, 2504.11343].
- **Diffusion Models (Image and Biomolecular Generation):** RAFT and its extensions (e.g., iterative distillation from soft-optimal policies [2507.00445]) enable diffusion models to optimize outputs according to reward functions, even those that are non-differentiable, with improved stability and sample efficiency relative to on-policy RL.
- **Retrieval-Augmented Question Answering:** Retrieval-Augmented Fine-Tuning (also known as RAFT in separate works [2403.10131, 2409.17648, 2506.06500]) improves in-domain QA performance in resource-constrained environments and domains with scarce labeled data. RAFT enables LLMs and small models to internalize retrieval, improving factual accuracy and robustness to noisy or distractor documents.
- **Reasoning and Math Problem Solving:** Methods such as ReFT extend RAFT with RL-based optimization of chain-of-thought outputs, notably improving generalization in math tasks by enabling models to learn from multiple reasoning paths [2401.08967, 2505.02391].
- **Electronic Design Automation (EDA) and Secure Access:** RAFT can be combined with synthetic Q/A generation, secure context filtering, and retrieval fusion to produce domain-precise and privacy-aware LLM assistants [2506.06500].

## 5. Comparative Analysis and Empirical Results

RAFT is systematically compared with PPO, GRPO, DPO, and other RL-based fine-tuning methods across LLM and diffusion domains:
- **Performance:** RAFT matches or exceeds PPO and GRPO in reward alignment and output fluency on language and vision tasks when evaluated via both automated and human metrics [2304.06767, 2504.11343].
- **Efficiency:** RAFT’s supervised updates and model decoupling reduce memory and computation, as only one model (not multiple actor/critic/reward reference policies) need be loaded at a time.
- **Stability:** RAFT avoids RL-specific instability due to policy divergence, especially when regularized with a KL penalty term:
  \[
  \tilde{r}(x, y) = r(x, y) - \beta \log\left[ \frac{p_g(y | x)}{p_{G_0}(y | x)}\right]
  \]
- **Sample Efficiency:** GVM-RAFT and Proto-RM provide further benefits, requiring fewer training samples and yielding faster convergence [2406.06606, 2505.02391].
- **Limitations:** In the absence of negative sample training, RAFT’s entropy can collapse, reducing exploration and robustness over prolonged training, a factor partially mitigated in GRPO/Reinforce-Rej variants [2504.11343].

Empirically, RAFT-based systems achieve notable gains, such as a >30% accuracy increase on HotpotQA in multi-hop QA [2403.10131], 2–4× speedups in chain-of-thought optimization [2505.02391], and superior data efficiency in human feedback reward modeling (Proto-RM, TinyRM).

## 6. Extensions and Future Directions

- **Hybrid Losses and Differentiable Ranking:** Combining analytic gradients when rewards are differentiable (DRaFT) with ranking-based losses can provide a flexible architecture interpolating between supervised reward maximization and hard best-of-K filtering [2309.17400].
- **Dynamic Curriculum and Task Difficulty Scheduling:** Adaptive curriculum methods such as AdaRFT can be layered onto RAFT for further improvements in reasoning performance and convergence [2504.05520].
- **Secure and Domain-Specific Fine-Tuning:** RAFT can incorporate secure context filtering, domain-adaptive curricula, or synthetic Q/A generation to address domains with constrained data or privacy requirements [2506.06500].
- **Efficient Reward Modeling:** Deployment of bidirectional MLM-based reward models (TinyRM) and prototypical networks (Proto-RM) facilitates efficient preference modeling under strict compute and data constraints [2406.06606, 2507.09973].

## 7. Summary Table: RAFT Core Variants and Characteristics

| RAFT Variant          | Model Domain         | Selection Mechanism        | Sample Update Strategy       | Key Advantages                       |
|---------------------- |---------------------|---------------------------|-----------------------------|--------------------------------------|
| Canonical RAFT        | LLM, diffusion      | Best-of-K ranking         | Supervised (MLE on pos. only)| Simplicity, stability, efficiency    |
| DRaFT / DRaFT-K/LV    | Diffusion           | Differentiable reward     | Backprop through sampling    | Sample efficiency, flexible gradients|
| GVM-RAFT              | LLM (CoT)           | Dynamic prompt-specific   | Variance-minimizing update   | Faster convergence, allocation opt.  |
| Proto-RM/TinyRM       | Reward model        | Prototypical/MLM scoring  | Efficient reward estimation  | Data-efficient, scalable             |
| Retrieval RAFT (CRAFT)| LLM/RAG             | Fine-tune on retrieved QA | Adapters (LoRA/DoRA)         | Resource efficiency, QA fidelity     |
| ReFT                  | LLM (math/reasoning)| PPO (RL) on CoT           | On-policy RL on rewards      | Generalization, multi-path learning  |

RAFT thus embodies a class of techniques that streamline reward-based alignment for generative models, leveraging sample ranking and preference modeling for scalable, interpretable, and stable fine-tuning across diverse application domains.

Source: https://www.emergentmind.com/topics/reward-ranked-fine-tuning-raft