Optimal SFT-to-RL Transition

• Optimal SFT-to-RL Transition is the process of selecting SFT checkpoints based on minimal generalization loss and high Pass@k metrics to ensure effective initialization for RL.
• It employs adaptive scheduling, progressive data scaling, and gradient-based evaluations to balance model memorization with exploratory capacity.
• Joint optimization frameworks like BRIDGE and SASR enhance the transition by dynamically blending SFT guidance and RL updates to mitigate catastrophic forgetting.

Optimal SFT-to-RL Transition

Optimal transition from Supervised Fine-Tuning (SFT) to Reinforcement Learning (RL) in post-training LLMs involves sophisticated strategies for metric-based checkpoint selection, curriculum design, progressive data scaling, and joint optimization objectives. In reasoning-intensive LLM and multimodal agent domains, naive reliance on post-SFT accuracy or "highest SFT score" is frequently misleading; effective SFT-to-RL transition requires approaches grounded in generalization loss, multi-sample evaluation metrics such as Pass@k, adaptive scheduling, and theoretical principles linking SFT consolidation and RL-driven capacity expansion (Kang et al., 2 Oct 2025, Ding et al., 12 Dec 2025, Yoshihara et al., 11 Jul 2025, Zhao et al., 12 Jan 2026).

1. Evaluation Metrics and SFT Checkpoint Selection

Optimal SFT-to-RL initialization is best determined by evaluating generalization loss on held-out reasoning examples and large-k Pass@k metrics, rather than maximizing raw post-SFT accuracy. Generalization loss $L_{\mathrm{gen}}(\theta_{\mathrm{SFT}})$ is defined as the mean negative log-likelihood (cross-entropy) of gold reasoning chains or answers on a held-out validation set $V$:

$$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$

where $\ell$ is the standard cross-entropy. Complementary to this, Pass@$k$ measures the probability that at least one of $k$ independently sampled outputs (chain + answer) is correct, providing finer-grained assessment of latent solution diversity—especially crucial for verifying RL latent "explorability":

$$\mathrm{Pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

where $c$ is the count of correct outputs among $n$ sampled outputs.

Empirical benchmarking in (Kang et al., 2 Oct 2025) demonstrates that post-SFT Pass@1 seldom explains more than $30\%$–$V$0 of RL outcome variance ($V$1), while minimum generalization loss and Pass@64 strongly predict post-RL ceiling (Spearman $V$2, $V$3 for Llama3-8B). Optimal transition is achieved by selecting the SFT epoch/checkpoint with the minimal $V$4 and highest Pass@64, prior to any evidence of overfitting as indicated by a rise in $V$5.

2. Data Budget, Example Diversity, and SFT Scaling Laws

Optimal SFT budget allocation favors repeated epochs on diverse halves of the data rather than single epochs over the full data. Overfitting via repeated exposure to a small homogeneous set artificially increases Pass@1 (homogeneous accuracy) but inflates $V$6, reducing subsequent RL headroom due to decreased exploration potential. Including examples of varying length and complexity—especially in mathematics and logical reasoning tasks—achieves better RL outcomes than training exclusively on the shortest or easiest instances.

Scaling laws established in (Ding et al., 12 Dec 2025) show that the post-training ceiling $V$7 scales linearly with log-compute dedicated to SFT and gains additional performance from increased trajectory difficulty (≈1.1×–1.2× multiplier). "Less is More" holds only for pure SFT; for post-training ceiling under SFT→RL, using more SFT data unambiguously improves outcomes.

SFT Regime	RL Headroom	Post-RL Ceiling	Comment
Underfitting	High	Low	RL can explore, poor SFT base
Stable (Optimum)	Moderate	High	Maximizes total performance
Mild Overfit	Low	Acceptable	Use if data is small/easy
Severe Overfit	Minimal	Degraded	RL gains collapse

3. Transition Scheduling: Metrics and Adaptive Workflows

Transition timing from SFT to RL should be grounded in empirical checks: stop SFT when validation $V$8 saturates at ≤2% above its minimum, or—when data is scarce—≤10% above minimum. In practice, monitor the trends of both $V$9 and Pass@large-k (e.g., Pass@64 or Pass@256) across checkpoints, launching RL from the checkpoint with minimal $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$0 or maximal Pass@k.

For multi-domain settings or dynamic datasets, utilize adaptive thresholding based on gradient-concentration metrics (Gini coefficient preferred), as operationalized in PRISM (Zhao et al., 12 Jan 2026). This splits the corpus into SFT-appropriate examples (diffuse gradients) and RL-appropriate cases (high gradient concentration), tuning the fraction routed to RL via an inverted-U-shaped validation curve. Default to a median split for RL allocation (≈50%), adjusting in 30–70% range as dictated by compute budget and model heterogeneity.

4. Joint and Adaptive SFT-RL Optimization Strategies

Recent work demonstrates that decoupled SFT→RL and RL→SFT can result in irreversible loss of either memorization (cross-entropy) or reward capacity (Niu et al., 12 Jan 2026). Bilevel cooperative frameworks such as BRIDGE (Chen et al., 8 Sep 2025) augment pure SFT→RL hand-off by maintaining a meta-learned component (e.g., LoRA) that adaptively weighs SFT guidance throughout RL training—maximizing cooperative gain and reducing catastrophic forgetting. Similarly, adaptive single-stage methods (SRFT (Fu et al., 24 Jun 2025), SASR (Chen et al., 19 May 2025)) balance SFT and RL at the mini-batch level, leveraging entropy or gradient-norm indicators to dynamically route updates.

Pseudocode snippet for gradient-norm-based switching (from SASR (Chen et al., 19 May 2025)): $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$6 This framework ensures continual, analytic, data-driven transition between SFT and RL regimes tailored to evolving model state and dataset properties.

5. Domain-Specific Regimes: Generative Classification, Vision-Language Agents, and Re-distillation

In generative classification tasks, the optimal SFT→RL transition is simply after "full warm-up" (entire SFT dataset converged), as further RL steps consistently yield additional accuracy (+3–6%) even when SFT baselines are high (He et al., 28 Apr 2025). Explicit chain-of-thought reasoning during RL is superfluous, with direct-answer prompts and accuracy-only rewards giving superior gains.

For vision-language-action agents utilizing flow-based models, a brief few-shot SFT pass on expert trajectories provides the necessary initialization for sample-efficient RL with exact Gaussian likelihoods (via Flow-Noise or Flow-SDE), as detailed in $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$1 (Chen et al., 29 Oct 2025). The recommended schedule is SFT (40–60 expert trajectories, freeze VLM), then RL in 64–320 simulations, achieving near-perfect generalization with only a few hundred RL epochs.

Re-distillation (Chen et al., 23 May 2025) offers a pragmatic regimen when compute is limited or RL sample effect plateaus: execute a short RL run to harvest ≈500–1,000 high-effect trajectory samples, then SFT on this distilled subset. This matches or exceeds full RL performance at a fraction of computational cost.

Protocol	Sample Efficiency	Peak Accuracy	Notes
Full SFT→RL	High	Best Overall	Costly, recommended default
PRISM Adaptive	Best Pareto	Higher OOD	1.76–3.22× compute speedup
Re-distillation	Max cost/eff	Matches RL	Only ≈1K samples required

6. Practical Transitional Workflows and Decision Criteria

An evidence-based optimal SFT-to-RL transition for reasoning LLMs is summarized in a stepwise workflow (Kang et al., 2 Oct 2025, Ding et al., 12 Dec 2025):

1. Design SFT candidates: Vary number of unique examples and epochs within compute budget.
2. Evaluate SFT: Compute held-out Pass@1, $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$2, Pass@large-k per checkpoint.
3. Filter/rank SFT checkpoints: Discard low-performance/rising $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$3; rank by Pass@large-k (or $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$4 if same distribution).
4. Predict RL outcomes: Optionally fit a regression from (Pass@large-k, $$L_{\mathrm{gen}}(\theta_{\mathrm{SFT}}) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_{\theta_{\mathrm{SFT}}}(y_i \mid x_i)\right)$$5) to actual RL performance on a subset.
5. Launch RL: Use top-ranked SFT as RL initialization.

For domains with intrinsic conflict or multimodal objectives, apply adaptive routing (PRISM), or meta-learned joint frameworks (BRIDGE, SRFT, SASR) to manage continual blending and reduce performance interference (Zhao et al., 12 Jan 2026, Chen et al., 8 Sep 2025, Fu et al., 24 Jun 2025, Chen et al., 19 May 2025, Niu et al., 12 Jan 2026).

Caveats to broad application include task-specific dynamics (e.g., code, multimodal tasks), cost of Pass@k estimation in long-sequence regimes, and the need for scaling law validation beyond mathematical reasoning. Direct outcome-based heuristics (validation accuracy) must be supplemented or replaced by intrinsic generalization and latent diversity metrics.

7. Limitations, Theoretical Coupling, and Future Directions

No purely sequential SFT→RL or RL→SFT scheme can preserve the optimality of both objectives; there exists an irreversible coupling whereby RL increases SFT loss, and SFT decreases RL reward (Niu et al., 12 Jan 2026). The minimal-loss SFT checkpoint is robustly optimal only when immediately followed by tailored RL with carefully calibrated sampling temperature (entropy ≈0.3) (Ding et al., 12 Dec 2025, Liu et al., 16 Jun 2025). Mixed or continual optimization via meta-gradient methods, gradient-concentration arbitration, or entropy-aware routing provides analytic and empirical performance guarantees in multi-stage pipelines.

Open directions include meta-learning dynamic mixing ratios, scaling transition metrics to new domains, and further formalization of the SFT–RL trade-off frontier in cross-domain, multi-task post-training (Niu et al., 12 Jan 2026, Zhao et al., 12 Jan 2026).