---
title: 'Two-Stage SFT + GRPO: LLM Optimization Pipeline'
url: https://www.emergentmind.com/topics/two-stage-sft-grpo-training-pipeline
type: topic
---

# Two-Stage SFT + GRPO: LLM Optimization Pipeline

A Two-Stage SFT + GRPO Training Pipeline is a systematic paradigm for optimizing large language models (LLMs) and multimodal neural architectures, designed to leverage the complementary strengths of Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). This approach has been rigorously validated as a practical and effective blueprint for domains such as mathematical reasoning, scientific information extraction, visual reasoning, recommendation systems, and medical AI, establishing new state-of-the-art results across diverse tasks. The central principle is to first maximize accuracy and task-specific structure via extended SFT, then apply GRPO to optimize efficiency or additional downstream desiderata, frequently with a composite reward function. This article provides a comprehensive technical description of the two-stage pipeline, focusing on its core methodology, algorithmic underpinnings, empirical best practices, and domain-generalization properties as established in recent literature.

## 1. Pipeline Structure and Motivation

The Two-Stage SFT + GRPO pipeline comprises:

- **Stage 1: Supervised Fine-Tuning (SFT):** The model is exposed to curated datasets of high-difficulty, annotated solution traces, with the objective of maximizing accuracy via next-token cross-entropy minimization. Careful control over dataset diversity, sequence length, and optimization regime (e.g., multi-epoch SFT, low learning rate, curriculum ordering) is critical for saturating performance.
- **Stage 2: Group Relative Policy Optimization (GRPO):** The SFT-tuned model is further trained with reinforcement learning, using reward functions that encode not only correctness but also token efficiency, output length, adherence to format, and other task-specific behavioral constraints. GRPO eschews value networks, instead employing intra-group comparisons to stabilize advantage estimation and policy gradients.

This two-stage recipe exploits the fact that SFT and RL serve non-redundant roles: a prolonged SFT phase pushes accuracy to its limit, while a gentle, KL-constrained GRPO phase enables the policy to compress and rationalize behaviors—for example, by shortening mathematical solutions without loss of correctness [2507.08267], or removing hallucinated items in recommendations [2510.20150].

## 2. Methodological Details

### 2.1 Supervised Fine-Tuning

The SFT phase trains the LLM on task-specific demonstration data $(x, y_{1:T})$, minimizing

\[
L_{\mathrm{SFT}}(\theta) = -\sum_{t=1}^T \log \pi_\theta(y_t \mid y_{<t}, x)
\]

Key SFT implementation features include:

- **Datasets:** Task-specific, high-quality traces (e.g., 7,900 hard math problems in [2507.08267]; 22M annotated visual data in [2510.12798]).
- **Batching and Tokenization:** Sequence packing enabled; long contexts supported (up to 24,000 tokens).
- **Optimization:** AdamW with initial learning rate $1\times 10^{-5}$, cosine decay; parameter-efficient LoRA or DoRA adapters for very large models or low-precision inference [2506.21594].
- **Prompt engineering:** Explicit reasoning templates and system prompts (such as "Please reason step by step, and put your final answer within \texttt{\textbackslash boxed\{\ldots\}}").
- **Epoch schedule:** Experiments show that extending SFT to $\sim$10 epochs is essential to push accuracy to its ceiling, especially on small, high-difficulty datasets [2507.08267].

### 2.2 Group Relative Policy Optimization (GRPO)

The GRPO stage takes the SFT model as initialization and proceeds as follows:

1. **Rollout Sampling:** For each prompt, generate a group of $N$ candidate solution traces.
2. **Reward Assignment:** Compute a scalar reward $R(\tau)$ for each trace based on a composite of:
   - Format adherence,
   - Semantic correctness (regex, answer verification),
   - Cosine similarity to reference embeddings [2507.08267],
   - Output length penalties or diversity/novelty [2510.20150].

The canonical reward formula is:

\[
R(\tau) = \alpha\, r_1(\tau) + \beta\, r_2(\tau) + \gamma\, r_3(\tau)
\]
where each $r_i$ targets a distinct behavioral dimension.

3. **Advantage Estimation:** Compute relative advantage for each trace within the group:

\[
\hat{A}_i = R(\tau_i) - \bar{R}_{\mathrm{group}}
\]

4. **Policy Update:** Solve the clipped surrogate with KL penalty:

\[
L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}_i\left[ \min(r_i(\theta)\,\hat{A}_i,\, \mathrm{clip}(r_i(\theta),1-\epsilon,1+\epsilon)\,\hat{A}_i) \right] 
+ \beta_{\mathrm{KL}}\ \mathrm{KL}(\pi_{\theta_{\rm old}}\,\|\,\pi_\theta)
\]
with $r_i(\theta) = \pi_\theta(\tau_i)/\pi_{\theta_{\rm old}}(\tau_i)$, and periodic $\theta_{\rm old} \leftarrow \theta$ [2507.08267].

## 3. Algorithmic Pseudocode

The following pseudocode (quoted from [2507.08267]) summarizes the canonical two-stage workflow:

```python
# Stage 1: SFT
θ ← base_model
for epoch in 1...10:
    for batch B in SFT data:
        L_sft ← -sum_{(x, y) in B} sum_{t=1}^{|y|} log π_θ(y_t|y_{<t}, x)
        θ ← θ - η_sft * ∇_θ L_sft

# Stage 2: GRPO
θ_old ← θ
for step in 1...50:
    sample mini-batch {x_j}
    for each x_j: generate N=8 samples τ_{j,1..8} ∼ π_θ
    compute rewards R_{j,i}
    group baseline bar_R_j ← mean_i R_{j,i}
    advantages hat_A_{j,i} ← R_{j,i} - bar_R_j
    compute GRPO loss as above
    θ ← θ - η_rl * ∇_θ L_GRPO
    periodically: θ_old ← θ
```

## 4. Empirical Evaluation and Ablations

Systematic evaluation and ablation across high-difficulty benchmarks demonstrate several key findings:

- **Extended SFT:** Accuracy on AIME 2024/2025 rises with SFT epochs, plateauing only at epoch 10; short SFT (1–3 epochs) increases solution length and destabilizes RL [2507.08267].
- **GRPO for Efficiency:** GRPO phase reduces solution length by $\sim$20% (e.g., from 10.3k to 7.9k tokens on AIME 2024 at 14B scale) without accuracy loss; in some cases, accuracy slightly improves post-GRPO [2507.08267].
- **Reward Engineering:** The combination of format, cosine similarity, and length penalty yields the optimal Pareto frontier (high accuracy, low token budget); dropping components degrades efficiency or accuracy [2507.08267].
- **Robustness:** KL-regularization prevents collapse; group size $N=8$ suffices for stable advantage estimation.
- **Generalization:** Recipe validated in domains beyond mathematics (see "MimicSFT + R²GRPO" for SciIE [2505.22068]), recommendations [2506.19235, 2510.20150], visual reasoning [2508.11196, 2502.14669], and medical reasoning [2506.21594].

## 5. Design Rationale and Best Practices

The pipeline's structure and hyperparameters are justified by quantitative and qualitative analysis:

- **Prolonged SFT:** Necessary for models to approach maximal accuracy given limited demonstration data; short fine-tuning phases induce under-specification and RL instability [2507.08267].
- **Gentle GRPO:** A restrained RL phase (e.g., 50 steps with low learning rate) targets efficiency and structural refinement, rather than overhauling accuracy.
- **KL Penalty:** Maintains proximity to SFT-initialized policy and prevents catastrophic divergence.
- **Reward Scaling:** Length penalty parameters are calibrated such that a $1\%$ increase in output length corresponds to a fixed decrement in total reward, balancing brevity and completeness.
- **System Prompts:** Explicit instructions such as "reason step by step" and "final answer within \boxed{}" anchor solution format and are critical for downstream verifiability.

## 6. Limitations, Generalizations, and Extensions

While the two-stage SFT + GRPO paradigm is robust across tasks, context-specific limitations and considerations have emerged:

- **Overfitting Risks:** Excessive SFT epochs or small, homogeneous datasets can produce overfit policies that suppress RL gains [2510.01624].
- **Dataset Composition:** Including a mix of example lengths in SFT, and emphasizing hard instances, enhances the effectiveness of subsequent RL [2510.01624].
- **Alternative Schedules:** Adaptive switching (cf. SASR [2505.13026]) between SFT and GRPO guided by training signal statistics can further improve stability.
- **Reward Misspecification:** Inadequate or ill-calibrated reward components can bias RL toward trivial or verbose outputs [2507.08267]; ablations confirm the necessity of component diversity.
- **Cross-Domain Applicability:** The pipeline generalizes beyond text and mathematics to structured visual reasoning, scientific IE, medical reasoning, and navigation [2505.22068, 2506.21594, 2508.11196, 2506.04070], often with task-specific adaptation of datasets, prompts, and rewards.

## 7. Conclusion and Future Directions

The Two-Stage SFT + GRPO Training Pipeline delivers a reproducible, high-performance recipe for end-to-end LLM optimization: extended SFT saturates accuracy, while KL-regularized GRPO selectively improves solution efficiency, format adherence, and other secondary metrics. This methodology has set new state-of-the-art results in the AI Mathematical Olympiad (AIMO) and diverse other tasks, and is supported by comprehensive open-source releases of code and checkpoints [2507.08267]. Continued research is likely to refine reward engineering, explore adaptive SFT–RL schedules, and expand the paradigm into increasingly complex domains—an approach supported by the robust empirical and theoretical footing documented to date.

Source: https://www.emergentmind.com/topics/two-stage-sft-grpo-training-pipeline