SFT Generalization Paradox

Updated 2 December 2025

SFT Generalization Paradox is the phenomenon where increasing in-distribution performance through supervised fine-tuning leads to a non-monotonic, often degraded, out-of-distribution accuracy.
Spectral analyses reveal that SFT reorients singular vectors in model weights, aligning them with in-domain targets while undermining pretraining-acquired, generalizable subspaces.
Mitigation strategies, including RL fine-tuning, spectrum-aware early stopping, and shallow-layer resets, can restore much of the lost OOD performance without sacrificing in-id gains.

The “SFT Generalization Paradox” refers to the counterintuitive phenomenon that increasing in-distribution (“ID”) performance of LLMs or vision-LLMs (VLMs) via supervised fine-tuning (SFT) often coincides with degraded out-of-distribution (“OOD”) generalization. While SFT continues to monotonically improve ID accuracy during training, OOD performance frequently peaks early, then deteriorates—sometimes catastrophically—as specialization progresses. Recent research has elucidated the underlying spectral, mechanistic, and practical causes, and has developed a set of spectrum-aware, curriculum, and reinforcement learning (RL) interventions that restore generalization without sacrificing in-domain gains or model capacity.

1. Formal Definition and Canonical Manifestations

The SFT Generalization Paradox is observed when supervised fine-tuning on a target task $\mathcal{D}_{\rm ID}$ monotonically increases in-distribution accuracy while OOD accuracy on novel variants or transfer tasks peaks at an intermediate checkpoint before declining with further fine-tuning. This dissociation is quantified as follows (Jin et al., 22 Aug 2025, Jin et al., 8 Sep 2025, Cheng et al., 1 Dec 2025):

Let $A_\text{ID}(t)$ $A_{ID} (t)$ and $A_\text{OOD}(t)$ $A_{OOD} (t)$ denote the ID and OOD accuracies after $t$ $t$ SFT steps.
- $A_\text{ID}(t)$ increases with $t$ .
- $A_\text{OOD}(t)$ increases initially (up to $t^*$ ), then decreases: $\max_t A_\text{OOD}(t) = A_\text{OOD}(t^*)$ , with $A_\text{OOD}(t \to \infty) \ll A_\text{OOD}(t^*)$ .
This pattern is robust across domains: LLMs on GeneralPoints card-game OOD variants (Jin et al., 22 Aug 2025, Jin et al., 8 Sep 2025), VLMs on multimodal reasoning benchmarks stratified by difficulty (Chen et al., 10 Jul 2025), domain-specific LLM SFT (Lin et al., 25 Sep 2025), and synthetic compositional reasoning tasks (Cheng et al., 1 Dec 2025).
Key symptoms include:
- SFT “forgets” pretraining-acquired OOD reasoning capacity while achieving near-perfect ID fit (“SFT forgetting”).
- Standard per-example train/test loss and ID metrics fail to signal the OOD optimum.
- Subsequent RL fine-tuning can restore—though not surpass—OOD performance if applied in the correct regime.

2. Spectral Mechanisms: Singular Vector Rotation and Subspace Drift

Recent studies employing singular value decomposition (SVD) have attributed the paradox not to reduction in weight matrix norm or singular value collapse, but to a rotation of singular vectors, i.e., the reorientation of representation subspaces critical for general reasoning (Jin et al., 22 Aug 2025, Jin et al., 8 Sep 2025):

Any transformer layer’s weight matrix $W^{(\ell)}$ $W^{(ℓ)}$ can be decomposed as $W^{(\ell)} = U^{(\ell)} \Sigma^{(\ell)} V^{(\ell)T}$ .
- SFT leaves singular values $\{\sigma^{(\ell)}_i\}$ nearly invariant, preserving overall capacity.
- The dominant change is a drift in the principal singular vector directions ( $\theta^{(\ell)}_i$ for top/bottom $i$ ), measured via cosine similarity or principal angles.
SFT reorients top and bottom singular vectors, aligning them with solutions optimal for $\mathcal{D}_{\rm ID}$ but destroying alignment with pretraining subspaces necessary for OOD generalization.
RL-FT, particularly via reward-driven PPO, can realign these singular vectors to their pre-SFT orientations, restoring most of the lost OOD performance but cannot recover if SFT-induced drift is too large (i.e., SFT was “overcooked”).
Practically, restoring just the top 20% of singular directions or resetting early layers recovers 70–80% of OOD loss, enabling interventions prior to expensive RL (Jin et al., 22 Aug 2025).

3. Empirical Phenomena Across Modalities and Tasks

Empirical analysis reveals modality-general and task-specific signatures (Jin et al., 8 Sep 2025, Jin et al., 22 Aug 2025, Chen et al., 10 Jul 2025, Chu et al., 28 Jan 2025, Cheng et al., 1 Dec 2025):

On LLMs for reasoning (e.g., Llama-11B, Qwen-7B), SFT on $\mathcal{D}_{\rm ID}$ $D_{ID}$ :
- OOD accuracy may drop from 17–20% (max SFT) to 9–17% (after continued SFT).
- RL-FT restores to near the OOD peak, but never exceeds it (Jin et al., 22 Aug 2025, Jin et al., 8 Sep 2025).
In VLMs, SFT on long chain-of-thought (CoT) traces for hard questions improves accuracy for high-difficulty (L4/L5) subsets but reduces performance on simple questions (L1/L2) by forcing verbose, over-elaborate reasoning (Chen et al., 10 Jul 2025).
- RL recovers brevity and generalizes, improving or restoring performance across all difficulty levels.
On composite reasoning tasks (mixing memory/context hops), SFT on composite data alone leads to high in-distribution (IID) performance (90%) but catastrophic collapse on zero-shot OOD (18%). RL on a base pretrained for the atomic skills enables synthesis and compositional generalization (50% zero-shot), contingent on passing an atomic skill threshold (Cheng et al., 1 Dec 2025).

4. Diagnostic Metrics and Early Stopping

Standard SFT monitoring via training/test loss or held-in accuracy is insufficient and may be actively misleading in checkpoint selection for downstream generalization or RL (Jin et al., 8 Sep 2025, Kang et al., 2 Oct 2025):

OOD cross-entropy loss and accuracy can be non-monotonic and diverge from ID loss.
Singular vector principal angles ( $\theta_i$ ) are reliable indicators of OOD capacity loss.
Pass@k metrics at large $k$ (e.g., Pass@64), and generalization loss on held-out reasoning, are strong predictors for final RL outcomes.
Generalization loss and Pass@k (as opposed to Pass@1 or accuracy on easy data) correlate with post-RL potential (Spearman $\rho \approx$ 0.94–0.98).

Practitioners must evaluate checkpoints on representative OOD slices and/or generalization metrics and avoid selecting based on highest ID accuracy or loss reduction alone.

5. Mitigation Strategies: Spectral, Curriculum, and Proximal Methods

A plethora of practical mitigations have emerged to circumvent or repair SFT-induced OOD degradation:

RL-Based Restoration: RL-FT via PPO realigns dominant singular directions, restoring up to 99% of OOD accuracy if applied within a recovery window (i.e., not too early or too late in SFT) (Jin et al., 22 Aug 2025, Jin et al., 8 Sep 2025, Chu et al., 28 Jan 2025, Cheng et al., 1 Dec 2025).
Low-Rank Merging and Shallow-Layer Reset: Interpolating between SVD subspaces of the pretrained and SFT models for top singular directions, or resetting early layers, provides an inexpensive OOD restoration mechanism (Jin et al., 22 Aug 2025).
Spectrum-Aware Early Stopping: Stop SFT at or near the OOD accuracy peak, as signaled by singular vector drift metrics or OOD validation (Jin et al., 22 Aug 2025).
Learning Rate and Token-Adaptive Reweighting: Lower SFT learning rates systematically reduce generalization loss without sacrificing domain-specific gains; Token-Adaptive Loss Reweighting (TALR) downweights hard/rare tokens prone to over-specialization (Lin et al., 25 Sep 2025).
Proximal SFT: Incorporating trust-region or KL-divergence constraints (Proximal SFT) controls policy drift, stabilizes entropy, and preserves out-of-domain generalization, outperforming standard SFT on OOD tasks (Zhu et al., 25 Aug 2025).

6. Theoretical Perspectives: SFT Gradient Pathologies and RL Synthesis

The SFT objective admits a policy-gradient interpretation with constant rewards (behavior cloning), where the gradient is weighted by $1/\pi_\theta$ , yielding unbounded variance and overconcentration on rare tokens—mechanistically leading to memorization and spectral drift (Wu et al., 7 Aug 2025). RL methods, in contrast, supply informative reward-driven gradients that reinforce generalizable subspaces.

Dynamic Fine-Tuning (DFT), which dynamically rescales SFT gradients by output probabilities, eliminates these ill-posed weights and closes the generalization gap with RL via a trivial code change (Wu et al., 7 Aug 2025).
In composite reasoning, RL can “synthesize” new multi-hop compositional strategies from atomic subskills, but only if atomic performance surpasses a threshold—otherwise, RL merely amplifies memorized SFT pathways (Cheng et al., 1 Dec 2025).

7. Implications for Foundation Model Post-Training and Open Directions

The SFT Generalization Paradox challenges the assumption that in-domain gains transfer to general capability and highlights the fragility of pure SFT for reliable foundation model adaptation. The emerging best practice is a spectrum-aware, multi-stage pipeline:

SFT for format and atomic skill acquisition (with careful regularization and monitoring).
Early stopping or spectrum/validation-driven checkpoint selection.
RL-FT (e.g., PPO, GRPO) for directional realignment and OOD restoration.
Spectrum-aware merging or shallow resets as inexpensive alternatives.
Continual validation on diverse OOD benchmarks, supplemented by entropy and singular vector diagnostics.

The paradox’s resolution points toward the need for difficulty-adaptive objectives, curriculum-aware training, and further theoretical work on the geometry of parameter-space evolution during SFT versus RL. Open problems remain in defining optimal trust regions, quantifying singular vector drift in very large models, and designing unified objectives that retain depth on hard tasks without sacrificing brevity and robustness on simpler or shifted distributions (Chen et al., 10 Jul 2025, Zhu et al., 25 Aug 2025, Jin et al., 22 Aug 2025).