---
title: Sequential SFT-then-RL Pipeline
url: https://www.emergentmind.com/topics/sequential-sft-then-rl-pipeline
type: topic
---

# Sequential SFT-then-RL Pipeline

A sequential Supervised Fine-Tuning-then-Reinforcement Learning (SFT-then-RL) pipeline is the prevailing methodology for post-training large language models (LLMs) and multimodal agents to achieve advanced task alignment and reasoning capabilities. This pipeline first relies on SFT to inject task-specific knowledge, followed by RL for performance adaptation via reward-driven optimization, often with additional regularization to preserve the benefits of SFT. The paradigm is characterized by its modularity, stability, and high empirical ceilings, and has been implemented in state-of-the-art systems across code, reasoning, vision-language, and generative model domains.

## 1. Pipeline Structure and Workflow

A canonical sequential SFT-then-RL pipeline consists of three core stages:

1. **Data Curation**: Automated or semi-automated selection and filtering of high-quality supervision traces from sources like curated code repositories, synthetic reasoning datasets, or procedure-distilled model outputs. Techniques such as SPICE-based filtering and rejection sampling may be employed to ensure clarity, difficulty-appropriate, and coverage-filtered data [2508.01550].

2. **Supervised Fine-Tuning (SFT)**: The model $\pi_\theta$ is initialized and optimized via cross-entropy minimization on the chosen dataset,
   $$
   L_\text{SFT}(\theta) = -\mathbb{E}_{(x,y)\sim\mathcal{D}_\text{SFT}} \left[ \sum_{t=1}^T \log \pi_\theta(y_t \mid y_{<t}, x) \right],
   $$
   typically for a fixed number of epochs and batch size, with checkpoint selection governed by validation loss.

3. **Reinforcement Learning (RL)**: The SFT-initialized policy is further refined to maximize reward through policy gradient algorithms (e.g., PPO, GRPO), optionally regularized by KL-divergence to the SFT reference,
   $$
   J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)] - \lambda \mathrm{KL}(\pi_\theta(\cdot) \| \pi_\text{ref}(\cdot)),
   $$
   where $R(\tau)$ is a verifiable reward (e.g., passing all test cases, correct answer extraction) and $\lambda$ sets the regularization strength [2508.01550, 2506.17211, 2507.08267].

This pipeline supports fully automatic, scalable deployment by orchestrating data ingestion, supervised optimization, and distributed asynchronous RL—often using frameworks like Ray for rollout parallelism [2508.01550]. The full system is documented by task-specific pseudocode and hyperparameter settings that ensure reproducibility.

## 2. Mathematical Formulation and Objectives

The SFT phase minimizes cross-entropy over high-quality demonstration trajectories, instilling prior knowledge and format regularization. For sequence models,
$$
L_\text{SFT}(\theta) = -\sum_{i=1}^N \sum_{t=1}^{T_i} \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x^{(i)}).
$$

The RL phase maximizes the expectation of a verifiable, often sparse, reward signal,
$$
J_\text{RL}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)] - \lambda \,\text{KL}(\pi_\theta(\cdot)\|\pi_\text{ref}(\cdot)).
$$
Policy optimization is implemented via PPO-style or group normalizing surrogates, exploiting advantage estimators and KL-penalties to ensure stable policy updates. For example, in KL-regularized PPO,
$$
g(\theta) \approx \mathbb{E}_{\tau\sim\pi_\theta}\left[ \sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot (R(\tau) - b) - \lambda \nabla_\theta \mathrm{KL}(\pi_\theta(\cdot|s_t) \| \pi_\text{ref}(\cdot|s_t)) \right],
$$
with baseline $b=0$ in binary-reward settings.

In multistage settings, the SFT loss is set to zero once RL commences, though optional mixed gradients, $L_\text{total} = \alpha L_\text{SFT} + (1-\alpha) L_\text{RL}$ for small $\alpha$, can be used during early RL steps [2512.12690].

## 3. Empirical Results and Scaling Laws

Empirical studies consistently demonstrate that sequential SFT-then-RL pipelines outperform pure SFT or pure RL, especially when staged according to validation loss and data scaling principles:

| Model                   | Accuracy (%) SFT | Final Acc. (%) SFT→RL | Reference                |
|-------------------------|------------------|-----------------------|--------------------------|
| Code LLM (SWE-Bench-V)  | 12.7             | 17.4                  | [2508.01550]             |
| VLM (Qwen2-VL-7B)       | 47.7             | 50.2                  | [2512.12690]             |
| Math LLM (AIME 14B)     | 65.2             | 66.0                  | [2507.08267]             |

Key patterns include:
- SFT boosts performance by establishing a task-appropriate initialization, often producing steeper gains for lower-capacity models or under extended data regimes [2512.12690, 2506.13284].
- RL applied to an SFT-warm start further improves the ceiling by exposing the policy to distributional feedback, verifiable signals, or task-specific brevity penalties [2507.08267].
- Data scale and trajectory difficulty in SFT directly determine both the “foundation” and the “plasticity” available for RL; selecting the best SFT checkpoint (minimum validation loss) before RL maximizes the empirical ceiling [2512.11470].

## 4. Scheduling, Hyperparameters, and Best Practices

Comprehensive guidelines have been formalized to maximize final performance:

- **SFT Stage**: Train until validation loss saturates in the “Stable” or at most “Mild Overfitting” regime, using large, high-quality SFT datasets. Data diversity and difficulty curation act as critical multipliers [2512.11470], with prompt diversity contributing more strongly than response multiplicity [2506.13284].

- **Transition Criterion**: Switch to RL when SFT validation loss reaches its plateau (within 2% of its minimum), avoiding severe overfitting which collapses RL plasticity [2512.11470].

- **RL Stage**: Use KL coefficients in the range $\lambda \in [0.001, 0.04]$ (task-dependent) to balance reward pursuit and policy stability; select batch sizes and rollout counts proportional to task and model scale (e.g., batch=32, G=8–10 rollouts/thread, RL steps ∼50–200) [2507.08267, 2508.01550]. Sampling temperature during RL should maintain entropy (e.g., $H_T(\pi)\approx 0.3$ in “nats”) to avoid premature convergence [2506.13284].

- **Data Mix and Curriculum**: For code and reasoning, mixing atomic “building block” tasks and composite samples ensures generalization, while staged exposure to length or difficulty-crafted samples serves as a curriculum [2406.10305, 2506.17211].

- **Checkpoint Selection**: Always pass the minimum validation-loss SFT model to RL. Monitor held-out accuracy during RL, employing early-stopping if reward increases without accuracy gains (to avoid “deceptive rewards”) [2512.12690].

## 5. Limitations, Failure Modes, and Extensions

Sequential SFT-then-RL pipelines face several documented limitations:

- **Catastrophic Forgetting**: Pure RL phases following SFT can rapidly “forget” behaviors acquired by SFT, especially in the absence of ongoing supervised guidance or properly tuned KL penalties [2509.06948].
- **Expressiveness and Sparse Reward Trap**: The SFT phase may provide no useful supervision if the expert traces are out of reach for a lower-capacity model, leading to RL failures due to absent reward signals. Methods such as BREAD anchor rollouts on partial expert prefixes to densify rewards and avoid this barrier [2506.17211].
- **Pseudo Reasoning Lock-In**: In multimodal and VLM settings, heavy SFT on expert reasoning traces may trap the policy into “pseudo reasoning” forms—mimicking structure, but not achieving authentic generalization or improvement under subsequent RL [2504.11468].
- **Tuning vs. Generality Tradeoff**: Over-specialized SFT phases can impair out-of-domain robustness; interventions such as RL-from-scratch or atomic-only RL can help recover generalization for code and language models [2406.10305].

Algorithmic extensions include per-instance adaptive pipelines (SuperRL) that fall back to SFT whenever RL fails to elicit positive reward signals [2506.01096], stepwise adaptive blending (SASR) with dynamic SFT/RL mixing based on training dynamics [2505.13026], and bilevel cooperative SFT-RL that meta-learns the optimal degree of SFT retention through upper-level optimization [2509.06948].

## 6. Application Domains and Empirical Benchmarks

Sequential SFT-then-RL pipelines are foundational in a wide range of domains:

- **Code LLMs/SWE**: Pipelines such as RepoForge use autonomous curation, SFT on multi-turn code repair trajectories, and KL-regularized RL to achieve state-of-the-art on SWE-Bench-Verified [2508.01550].

- **Mathematical and Logical Reasoning**: Frameworks for mathematical LLMs (e.g., AIME, MATH500, LightR1) deploy extended SFT and token-efficient RL (GRPO) to maximize reasoning accuracy and brevity [2507.08267, 2506.13284].

- **Vision-Language Models (VLMs)**: Qwen2-VL and similar architectures use rejection-sampled expert traces for SFT and group-based policy optimization for RL, achieving improved cross-modal generalization and data efficiency [2512.12690].

- **Autoregressive Generative Models**: ReasonGen-R1 injects explicit reasoning as textual rationales before image token generation, with SFT followed by RL on visual quality rewards to robustly improve compositionality and image alignment [2505.24875].

- **Instruction-Following**: Pipelines combining SFT and RL with supervised reward (RLSR) achieve superior instruction adherence and open-ended generation quality, outperforming SFT-only baselines [2510.14200].

- **Hybrid and Adaptive Approaches**: SuperRL and SASR incorporate real-time SFT-RL switching for sparse reward robustness and convergence acceleration [2506.01096, 2505.13026].

## 7. Historical Evolution and Future Directions

The sequential SFT-then-RL pipeline has solidified as the community’s “default” post-training protocol for advanced LLM alignment, reasoning, and agentic behavior. Empirical studies such as those codified by the “Plasticity-Ceiling Framework” [2512.11470] provide explicit scaling law guidance and checkpointing rules to maximize task ceiling while preserving update plasticity, with extensive ablation validating these principles across domains and architectures.

Contemporary research explores adaptive blends, episode-anchored branching, and meta-optimization to overcome the rigidity and sample inefficiency of the naive sequential approach, leading to more data/compute-optimal and robust learning. Notably, as models and task definitions diversify, attention has shifted to the interface between SFT trace complexity, RL reward sparsity, and model capacity—highlighting the need for more nuanced, dynamical, and data-driven control of the SFT→RL transition and the integration of trajectory anchors or meta-learners to unlock further gains [2506.17211, 2509.06948].

The sequential SFT-then-RL pipeline thus serves as both a benchmark and a launchpad for new research into scalable, stable, and efficient post-training of LLM-based agents across modalities and domains.

Source: https://www.emergentmind.com/topics/sequential-sft-then-rl-pipeline