---
title: Two-Stage SFT-RL Pipelines
url: https://www.emergentmind.com/topics/two-stage-sft-rl-pipelines
type: topic
---

# Two-Stage SFT-RL Pipelines

A two-stage SFT→RL pipeline is a post-training regimen for large language models (LLMs) and related architectures that first applies Supervised Fine-Tuning (SFT) on curated demonstrations before online or on-policy Reinforcement Learning (RL) with respect to a task-aligned reward signal. These pipelines operationalize a principle of initialization and refinement: SFT leverages existing expert data to rapidly establish valid solution modes and stabilize learning, while RL subsequently sharpens performance by incorporating reward-driven exploration, solution diversity, and distributional shift handling. This approach has become foundational in both language-only and multimodal (e.g., vision-language) model alignment for complex reasoning tasks, coding, mathematical problem-solving, and other high-value AI capabilities.

## 1. The Standard Two-Stage SFT→RL Framework

The canonical SFT→RL pipeline consists of two sequential stages:

**1. Supervised Fine-Tuning (SFT) Stage:**  
A pretrained backbone model is fine-tuned on a dataset of high-quality, human- or LLM-generated prompt–response pairs, typically including chain-of-thought (CoT) reasoning traces for tasks necessitating stepwise inference. The SFT optimizes the autoregressive negative log-likelihood loss:
\[
\mathcal{L}_{\rm SFT}(\theta) = -\,\mathbb{E}_{(x,y)\sim\mathcal D}\bigl[\log \pi_\theta(y\mid x)\bigr]
\]
Here, SFT provides a strong inductive anchor for the policy, ensuring non-trivial initial solution rates even in environments with sparse rewards [2603.13985][2507.08267].

**2. Reinforcement Learning (RL) Stage:**  
The SFT-trained model is further adapted via RL to maximize expected reward, where the reward is typically based on explicit correctness, verifier approval, or human-preference modeling. RL updates are performed via on-policy algorithms such as PPO or Group Relative Policy Optimization (GRPO). The generic clipped PPO/GRPO objective is:
\[
\mathcal{L}_{\rm RL}(\theta) = -\,\E_{\tau\sim\pi_\theta}\bigl[R(\tau)\,\nabla_\theta\log\pi_\theta(\tau)\bigr] - \beta\,\mathrm{KL}(\pi_\theta\|\pi_{\rm ref})
\]
The KL penalty regularizes the updated policy against the SFT initialization to prevent drift, language collapse, or reward hacking [2603.13985][2506.01096].

The sequential pipeline can be extended with curriculum splitting (e.g., Decoupling SFT on easier instances from RL on harder ones [2603.11193]), adaptive mixing or fallback mechanisms [2506.01096][2505.13026], or explicit curriculum learning constructs.

## 2. Motivations and Theoretical Foundations

Empirical and analytical results consistently demonstrate that SFT and RL are complementary. SFT excels in sample-efficient acquisition of domain and format knowledge, driving rapid gains in basic solution accuracy and stabilizing early learning. However, pure SFT overfits to demonstration support and cannot incentivize off-distribution exploration or self-reflective correctness detection [2601.01580][2509.21128]. RL, in contrast, is necessary to concentrate probability mass on optimal behaviors, correct failure modes not covered by demonstrations, compress solution traces, or discover alternative strategies [2507.08267][2506.13284]. RL gradients are essential for optimizing decision components such as STOP/RESAMPLE selectors underpinning self-reflection in LLM policies [2601.01580].

Recent theoretical work formalizes this complementarity through the "Gradient Attribution Property" and related analyses. SFT objectives induce highly biased, low-variance updates, but collapse reward attribution to generative tokens, failing to optimize decisional policies. RL surrogate rewards (group-normalized advantages), in contrast, provide balanced gradients that simultaneously update sampling and verification policies, undergirding the emergence of error-detection and self-correction [2601.01580][2509.21128].

## 3. Advanced Variants and Extensions

Recent research introduces a variety of architectural and algorithmic extensions to the vanilla two-stage scheme to address known limitations:

- **Curriculum and Difficulty Decoupling:**  
  DeReason proposes splitting the training data into SFT and RL subsets via difficulty scoring, allocating non-reasoning-intensive examples to SFT and challenging ones to RL, improving both sample efficiency and overall generalization [2603.11193].

- **Dynamic or Adaptive Mixing:**  
  Frameworks such as SuperRL [2506.01096] and SASR [2505.13026] adaptively switch or mix between SFT and RL within the training loop, using reward feedback or gradient statistics (e.g., norm ratios) to select the objective per instance or minibatch, stabilizing training and ensuring learning signal coverage even when reward is sparse.

- **Reward Learning in SFT:**  
  Integrating inverse RL or preference-based reward learning into the SFT stage (RFT/IRFT [2405.17888]) can alleviate demonstration distribution shift and provide a regularizing signal that sharpens the model’s ability to distinguish high- and low-quality outputs.

- **Branched Rollouts and Partial Demonstration Injection:**  
  BREAD [2506.17211] supplements standard SFT→RL with expert prefixes ("anchors") whenever the model fails to generate correct traces, curricularly reducing reward sparsity.

- **Bias-Variance Tradeoff Reconciliation:**  
  DYPO [2604.08926] and related frameworks structurally combine variance-reduced RL objectives (e.g., group alignment loss) and bias-reduced SFT (e.g., multi-teacher KL distillation) with a dynamic reward-based gating system, demonstrating provable variance control and bias minimization.

- **Cooperative (Bilevel) SFT+RL:**  
  Bilevel schemes (BRIDGE) directly meta-train SFT to act as a “teacher” that efficiently guides RL, sidestepping catastrophic forgetting and promoting synergy through cooperative optimization [2509.06948].

## 4. Performance, Sample Efficiency, and Empirical Findings

Empirical results across language, code, mathematics, STEM reasoning, and vulnerability detection demonstrate that two-stage SFT→RL pipelines:

- Dramatically accelerate learning vs. RL from scratch, particularly in environments with sparse or hard-to-discover reward signals (e.g., OpenR1-Math, DeepScaleR) [2506.01096][2507.08267].
- Maximize both accuracy (via SFT) and token/computation efficiency (via RL): SFT pushes solution rate to the ceiling, while RL compresses reasoning traces and reduces sample redundancy [2507.08267][2506.13284].
- Improve cross-domain and cross-modal reasoning, especially in low- and moderate-capacity models and in low-data regimes; SFT’s sample efficiency reliably exceeds RL’s until larger scales, after which RL surpasses SFT in asymptotic accuracy [2512.12690].
- Are sensitive to SFT data quality and curriculum: excessive SFT ("over-SFT") can degrade RL’s exploration and plateauing performance, while insufficient SFT slows RL and increases reward hacking risk [2602.14012].
- Benefit from well-tuned RL sampling temperature and entropy to balance exploration and exploitation [2506.13284].
  
Quantitative highlights include:
- SuperRL achieves 75% EM on GSM8K in ∼200 steps vs ∼500 for PPO/GRPO; on sparse Math tasks, reward mean nearly doubles vs. vanilla RL [2506.01096].
- Difficulty-aware curricula (DeReason) yield +1.9% pass@1 over SFT-only and outperform both SFT→random RL and SFT-only on MMLU-Pro and GPQA-Diamond [2603.11193].
- Token efficiency (output length reduction) of up to 25–30% post-RL with preserved or improved accuracy in competitive mathematics [2507.08267].
- In large-scale benchmarks, SFT provides >10× data efficiency vs. RL, but RL outpaces SFT as the data budget and model capacity grow [2512.12690].

## 5. Challenges, Failure Modes, and Open Controversies

Despite the success of the SFT→RL paradigm, several limitations and active research directions remain:

- **Catastrophic Forgetting:**  
  Decoupled two-stage pipelines suffer from rapid loss of SFT-acquired behaviors at RL onset, evidenced by chain-of-thought length collapse and reward dips [2509.06948]. Bilevel and continued SFT supervision are proposed remedies.

- **Reward Sparsity and Imitation Bias:**  
  In hard reasoning domains, SFT-trained small models may have near-zero probability of discovering successful traces in RL, resulting in a lack of positive gradients (failure of the SFT+GRPO recipe). Branched rollouts and expert-anchored prefixes (BREAD) mitigate this [2506.17211].

- **SFT Overfitting and Exploration Stagnation:**  
  Excessive SFT training produces over-specialized policies unable to explore or self-improve under RL; hybrid or curriculum strategies address this [2602.14012][2507.07562].

- **Synergy Dilemma in Multimodal or Long-CoT Architectures:**  
  For VLMs, naively stacking SFT and RL often fails to yield additive benefits; SFT can worsen simplicity and RL may reduce reasoning depth—the "synergy dilemma." Adaptive, difficulty-aware scheduling and more nuanced data-model alignment are necessary [2507.07562][2512.12690].

- **Reward Model Aliasing and Deceptive Reward:**  
  RL agents may learn to maximize proxy rewards with no improvement or even regression in genuine reasoning accuracy; careful reward model design and SFT preconditioning are critical [2512.12690].

## 6. Practical Guidelines and Implementation Recipes

Practical two-stage SFT→RL pipelines require careful attention to the following elements:

| Stage      | Key Components                      | Typical Hyperparameters         |
|------------|-------------------------------------|--------------------------------|
| SFT        | Supervised NLL on (x, y*), CoT data | LR=1e-5, batch=32–128, 2–10 ep |
| RL         | GRPO/PPO with group advantages, KL  | LR=1e-6, K=5–16 rollouts, KL=1e-3–0.04, β=0.01–0.1, PPO clip=0.1–0.2 |

- Always ensure high-quality, distribution-aligned SFT data (avoid data leaks or imitative overfitting).
- Moderate SFT—sufficiently long to stabilize but not so extended as to inhibit RL diversity—is advisable.
- In RL, implement KL regularization and monitor both reward and held-out accuracy to detect reward hacking.
- When possible, partition data curricularly with non-reasoning tasks for SFT and high-reasoning tasks for RL.
- Consider adaptive or hybrid pipelines (e.g., SuperRL, SASR, DYPO) for domains with severe reward sparsity or instability [2506.01096][2505.13026][2604.08926].
- In small models, inject partial expert hints to bridge expressivity gaps [2506.17211].
  
Open-source implementations are provided for several recent pipelines (e.g., [AceReason-Nemotron-1.1-7B](https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B), [Kaggle-AIMO2-Fast-Math-R1](https://github.com/analokmaus/kaggle-aimo2-fast-math-r1)).

## 7. Outlook and Future Directions

Two-stage SFT→RL pipelines remain the dominant framework for LLM reasoning, alignment, and complex generative tasks. In parallel, the field is converging toward more unified or adaptive recipes that dynamically arbitrate between SFT and RL objectives, directly optimize curriculum allocations, and supplement reward-shaping with meta-learning and multi-objective criteria. Balancing efficiency, stability, and generalization—especially under distributional shift and reward sparsity—continues to motivate algorithmic advances. Analytical tools for credit assignment, trajectory analysis, and entropy regulation are increasingly deployed to understand and close remaining gaps between human-level and model-level reasoning [2509.21128][2601.01580].

In summary, the two-stage SFT→RL pipeline is both theoretically grounded and empirically validated as an effective, extensible paradigm for high-accuracy, robust, and efficient LLM post-training. Its continued evolution is central to the progress of scalable, generalizable, and trustworthy AI [2603.13985].

Source: https://www.emergentmind.com/topics/two-stage-sft-rl-pipelines