---
title: Reasoning SFT Stage Overview
url: https://www.emergentmind.com/topics/reasoning-sft-stage
type: topic
---

# Reasoning SFT Stage Overview

A Reasoning SFT (Supervised Fine-Tuning) Stage denotes the phase in large language model (LLM) or vision-language model (VLM) training where models are fine-tuned using datasets containing explicit reasoning demonstrations—often structured as natural language chain-of-thought (CoT) traces, stepwise rationales, or formal solution procedures. The goal is to directly impart procedural and compositional reasoning by maximizing the likelihood of human-annotated, interpretable reasoning trajectories. The Reasoning SFT stage is now a principal component in modern instruction-following, mathematical, scientific, and multimodal LLMs, and its methodology, limitations, and interactions with subsequent reinforcement-based stages define the effectiveness and generalizability of advanced reasoning models.

## 1. Core Principles and Objectives

The Reasoning SFT stage follows unsupervised pretraining or multi-modal pretraining and is designed with several objectives:

1. **Instruction/Format Alignment**: By training on large collections of (prompt, reasoning, answer) pairs, models are aligned to produce human-like, interpretable reasoning traces—often using a fixed response format such as
   ```
   <think> ... reasoning ... </think> <answer> ... </answer>
   ```
   [2503.20752][2507.14683][2504.11741][2510.14200].

2. **Inductive Bias Towards Structured Reasoning**: Explicit reasoning supervision biases models to produce stepwise decompositions and explanations, rather than single-step or surface-level answers [2503.20752][2504.11741][2510.12643].

3. **Activation of Latent Reasoning Capabilities**: SFT “activates” the model’s inherent, but unexpressed, reasoning, providing the policy initialization necessary for efficient reinforcement learning or downstream reward-driven adaptation [2503.20752][2509.03321].

4. **Teacher-Forcing with Cross-Entropy Loss**: The training objective is the token-level conditional log-likelihood:
   $$
   \mathcal{L}_\mathrm{SFT}(\pi_\theta) = - \sum_{i=1}^N \log \pi_\theta(y_i \mid x, y_{<i})
   $$
   with teacher-forcing on the correct tokens throughout each training sequence [2510.14200][2507.14683].

## 2. Methodologies and Dataset Construction

Datasets and implementation details crucially shape the efficacy of the SFT stage.

- **Data Sources and Annotation**:
  - **Human-annotated**: Curated, high-quality chain-of-thought traces, rationales, or deliberative answers, often costly to annotate at scale [2503.20752][2510.12643].
  - **Synthetically generated**: LLMs (using prompt templates or bootstrapping) generate chain-of-thought for new inputs; these are filtered or verified before use [2507.14683][2510.14200][2504.11741].
  - **Pattern-based**: For patterned reasoning tasks, it suffices to annotate generic reasoning patterns and use LLMs to generate rationales according to the template (PARO), enabling annotation scaling with minimal human labor [2510.12643].

- **Reasoning Trace Structure**:
  - Detailed, stepwise, and modular (e.g., in mathematical SFT: multi-step deduction, verification, error-checking; in VLMs: visually grounded, CoT explanations).
  - In $E^2C$-style paradigms, SFT data is decomposed into \((\text{plan}, \text{execution})\) with enforced plan-adherence during execution [2509.23946].

- **Size and Diversity**:
  - Reasoning performance sharply increases with SFT dataset scale, especially for complex problems [2504.11741][2507.08267].
  - For mathematical LLMs, extended SFT (8–10 epochs) and large, diverse corpus (e.g., 719K+ examples in MiroMind-M1) are critical for state-of-the-art accuracy [2507.14683][2507.08267].

- **Quality Control and Filtering**:
  - Aggressive filtering by reward models and rule-based validation, with decontamination from test sets, is now standard [2507.14683][2507.19748].
  - Token-packing can be eschewed for no-packing (better for long CoT) [2507.14683].

## 3. Empirical Effectiveness and Limitations

The direct effects (measured in diverse benchmarks) and remaining challenges of reasoning SFT are:

- **Accuracy and Generalization**:
  - Enables significant accuracy gains over pretraining-only models on medium-difficulty reasoning (e.g., math, code) with even small-scale SFT on R1-style traces [2504.11741].
  - For “patterned reasoning tasks,” reasoning *pattern* exposure suffices—annotation scale becomes sub-dominant [2510.12643].

- **Trajectory Expansion vs. Compression**:
  - SFT increases the diversity and number of correct reasoning trajectories (“trajectory expansion”) [2509.21128].
  - However, it also preserves diverse incorrect trajectories, so pass@1 accuracy may not improve without RL, but best-of-$k$ performance rises [2509.21128].

- **Foundational for RL/Reward Optimization**:
  - SFT is necessary for stable and sample-efficient RL by providing an effective initialization that avoids reward sparsity and degenerate optimization (cold-start RL is generally much less effective) [2503.20752][2507.14683][2509.03321][2507.08267].

- **Overfitting and OOD Forgetting**:
  - OOD reasoning peaks early during SFT and then declines with further training—a form of OOD forgetting that is not visible in standard validation loss [2509.12235].
  - Heavy/extended SFT can “lock in” imitative, rigid reasoning patterns, especially in vision-language models, making further RL less effective or even detrimental [2504.11468][2508.11196].

- **Expressivity Gaps**:
  - For very small models, SFT may be harmful if expert traces are too complex—SLMs may not even be able to imitate them, impeding subsequent RL [2506.17211].

## 4. Comparisons and Integration with Reinforcement Learning

SFT and RL have distinct, often complementary, influences on model reasoning:

- **Exploration vs. Determinism**:
  - SFT’s teacher-forcing precludes exploration—the model only sees ground-truth continuations, so cannot discover alternative paths or correct errors outside the labeled space [2510.14200].
  - RL introduces exploration by rewarding semantically correct but structurally diverse solutions—even if not identical to SFT labels.

- **Policy Distribution Effects**:
  - SFT acts as a “sledgehammer,” globally shrinking output entropy and enforcing high probability on target tokens everywhere [2506.19767][2509.21128].
  - RL acts as a “scalpel,” modifying only high-entropy (uncertain) states and selectively refining the policy [2506.19767].

- **Two-Stage and Single-Stage Schemes**:
  - Most strong models employ two-stage SFT$\rightarrow$RL pipelines for best performance (SFT for capacity, RL for optimality and robustness) [2507.08267][2510.14200][2503.20752][2509.03321].
  - Hybrid/single-stage approaches (SRFT) dynamically weight SFT and RL losses based on entropy, seeking to balance global structure with local exploration [2506.19767].

- **Quantitative Gains**:
  - RL (e.g., RLSR) with semantic rewards outperforms SFT on instruction following; hybrid SFT+RLSR further boosts open-ended, generative performance (AlpacaEval win rates: SFT 21.0%, RLSR 26.3%, SFT+RLSR 30.7% on Qwen-7B) [2510.14200].

## 5. Data Design, Pattern Supervision, and Emerging Paradigms

Data design strongly determines SFT effectiveness, generalization, and scalability.

- **Patterned Tasks and Rationale Automation**:
  - For tasks with a fixed procedural pattern (classification, verification), minimal annotated rationales plus pattern-guided LLM rationale generation suffice (PARO: SFT+RLVR with LLM-generated rationales matches 10$\times$ human rationale scale) [2510.12643].

- **Plan-Execution Decoupling**:
  - SFT can be structured to output plans and executions in a causally-adhered, separated manner ($E^2C$), enabling far more efficient reasoning, cross-domain transfer, and interpretability [2509.23946].

- **Selective and Efficient Reasoning**:
  - SFT methodology can encode selective behaviors (e.g., “thought dropout” to allow the model to skip reasoning on easy problems, unlocking significant efficiency in VLMs) [2505.16854].

- **Data Quality over Quantity**:
  - For mathematical and code reasoning, SFT dataset *quality*—i.e., the correctness, clarity, and CoT depth of reasoning traces—trumps raw scale; scaling SFT in low quality, mixed data regimes may harm reasoning [2510.03264][2504.11741].

## 6. Limitations and Best Practices

- **Ceiling Effects**:
  - SFT alone propagates reasoning up to a “Hard” tier of task difficulty, with accuracy plateauing (e.g., $\sim$65% for hard AIME24 problems), regardless of further SFT scaling or curation [2504.11741].
  - Exceptional, out-of-domain, or “unconventional” problem solving generally requires new architectural, program-augmented, or externally conditioned training paradigms.

- **SFT as Foundation, Not Panacea**:
  - SFT is necessary but not sufficient for state-of-the-art reasoning: without subsequent RL, semantic reward optimization, or plan-execute architecture, SFT-imparted skills are limited to imitative, rigid, or failure-prone modes on complex, compositional reasoning [2510.14200][2509.21128][2504.11468].
  
- **Ongoing Directions**:
  - Optimization of SFT trajectories for variety and coverage, rather than correctness alone, and closer integration of entropy diagnostics to regulate SFT/RL phases are emerging as effective methodologies [2509.21128][2506.19767].
  - Pattern-based rationale annotation, causally-structured plan-execute data generation, and difficulty stratification via teacher models (for ZPD) are shaping future scalable, automated SFT pipelines [2509.23946][2510.12643][2509.03321].

---

**Summary Table: SFT Stage Roles and Trade-offs**

| Aspect                | SFT Stage Contribution            | Limitation/Trade-off                      |
|-----------------------|-----------------------------------|-------------------------------------------|
| Alignment/Format      | Strong task and format imitation  | Rigid reasoning, limited exploration      |
| Reasoning Expansion   | Broadens correct solutions space  | Retains diverse errors, not precise enough|
| Generalization OOD    | Peaks early SFT, then declines    | OOD forgetting with continued SFT         |
| RL Foundation         | Initializes stable policy for RL  | Can anchor model, reduce RL capacity      |
| Data Requirements     | Quality of CoT/plan pattern key   | Scale alone can plateau or harm           |

The Reasoning SFT stage thus remains indispensable for imparting interpretable, compositional reasoning to advanced LLMs and VLMs, but must be carefully designed for task structure, data scale/quality, procedural pattern representation, and later integration with reward-driven optimization to yield robust, domain-adaptive, and generalizable reasoning models.

Source: https://www.emergentmind.com/topics/reasoning-sft-stage