SFT+RL Protocol for LLM OOD Generalization

• SFT+RL protocol is a two-stage post-training process that uses supervised fine-tuning followed by RL fine-tuning to balance in-distribution and out-of-distribution performance.
• RL fine-tuning restores early OOD generalization lost during extended SFT, as evidenced by empirical benchmarks with models like LLaMA-11B and Qwen-7B.
• Monitoring singular vector rotations offers a precise diagnostic for identifying OOD forgetting, guiding the optimal policy handoff between SFT and RL.

A protocol combining Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)—hereafter "SFT+RL"—refers to post-training procedures for LLMs that sequentially or jointly optimize both imitation and task reward objectives. The SFT+RL paradigm is now canonical in LLM post-training, especially for eliciting complex reasoning, arithmetic, and generalization capabilities beyond those feasible with either paradigm alone. The SFT+RL protocol in "RL Fine-Tuning Heals OOD Forgetting in SFT" provides a tightly characterized, empirically validated framework for understanding, implementing, and diagnosing this two-stage post-training pipeline (Jin et al., 8 Sep 2025).

1. Paradigm Definition and Formalization

The standard SFT+RL protocol consists of two distinct fine-tuning stages applied to a pretrained LLM with parameter vector $\theta_0$:

(a) Supervised Fine-Tuning (SFT): Minimize negative log-likelihood (cross-entropy) over a labeled dataset $\mathcal{D} = \{(x_i, y_i)\}$: $$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$ Empirically O($10^3$) SFT updates are made, producing a sequence of checkpoints $\theta^{(t)}$. Early SFT (0–50 updates) aligns to task format; intermediate SFT (50–140 updates) develops arithmetic/reasoning abilities.

(b) RL Fine–Tuning: Initialize from an SFT checkpoint $\theta_{\text{SFT}}$, apply PPO to maximize expected task reward $R(x)$: $$L_{\text{PPO}}(\theta) = \mathbb{E}_t\big[\min\big(r_t(\theta)\,A_t,\, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\big)\big],$$ where $$r_t(\theta) = p_{\theta}(a_t|s_t)/p_{\text{old}}(a_t|s_t)$$ and $A_t$ is the advantage. Typical PPO convergence occurs within $\mathcal{D} = \{(x_i, y_i)\}$010 checkpoints.

The protocol is operationalized by first training SFT for several hundred steps, then applying PPO on the best available SFT checkpoint, where “best” is not necessarily determined by in-distribution (ID) accuracy or loss but by early peaks in OOD generalization.

2. OOD Forgetting and Restoration Phenomena

Out-of-Distribution (OOD) Metric

Generalization is measured in deliberately shifted evaluation conditions (e.g., a “GeneralPoints” card game with face cards J/Q/K remapped to 11/12/13). OOD accuracy is defined as: $\mathcal{D} = \{(x_i, y_i)\}$1

OOD Forgetting During SFT

Empirically, $\mathcal{D} = \{(x_i, y_i)\}$2 peaks extremely early (checkpoints 120–140), reaching $\mathcal{D} = \{(x_i, y_i)\}$3–$\mathcal{D} = \{(x_i, y_i)\}$4, after which it declines monotonically—termed "OOD forgetting"—even as ID metrics (loss, accuracy) continue to improve (ID accuracy $\mathcal{D} = \{(x_i, y_i)\}$5 at end-SFT). Critically, the ID loss or accuracy do not warn about this drop; format errors plateau after $\mathcal{D} = \{(x_i, y_i)\}$650 updates, and there is no sign of OOD capacity loss in standard validation diagnostics.

RL-Mediated OOD Restoration

RL applied to SFT checkpoints recovers OOD accuracy up to within $\mathcal{D} = \{(x_i, y_i)\}$7 of the early SFT OOD maximum, provided the SFT checkpoint is situated in a recovery window—empirically, SFT checkpoints with ID accuracy in $\mathcal{D} = \{(x_i, y_i)\}$8 ($\mathcal{D} = \{(x_i, y_i)\}$9 and $$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$0 updates for LLaMA-11B/Qwen-7B). RL does not deliver OOD generalization beyond the original SFT peak; its role is to restore previously lost capability rather than to synthesize fundamentally new generalization skills.

3. Protocol Boundaries, Empirical Guidance, and Failure Modes

The SFT+RL protocol is robust only within specific intervals of the SFT progression:

• SFT too short (<420 updates): The model's “base policy” is underfit, leading to reward hacking or RL collapse due to sparse positives.
• SFT too long (>1200 updates): Policy entropy collapses, causing RL advantage estimates to skew and PPO to stagnate, with no OOD recovery.

Practical protocol:

1. SFT for $$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$1–$$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$2 steps to learn format, continue to reach $$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$3–$$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$4 ID accuracy ($$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$5–$$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$6 steps).
2. Switch to PPO: batch size $$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$7256, clip $$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$8–$$L_{\text{SFT}}(\theta) = -\mathbb{E}_{(x, y)\sim \mathcal{D}}\, \log p_{\theta}(y|x).$$9, for $10^3$0 rollouts or until OOD accuracy plateaus (typically within $10^3$1–$10^3$2 of SFT OOD peak).
3. Monitor positive-to-negative reward ratio between $10^3$3 and $10^3$4.

Failures manifest as instability (RL from underfit SFT), reward hacking, or irrecoverable OOD loss (RL from overspecialized SFT).

4. Mechanistic Insights: SVD Analysis and Singular Vector Rotation

Parameter matrices $10^3$5 are decomposed via SVD: $10^3$6. Contrary to prior assumption, the singular values $10^3$7 remain nearly invariant through both SFT and RL ($10^3$8). However, substantial rotation occurs in the left/right singular spaces $10^3$9.

Quantitatively, principal angle spectra between $\theta^{(t)}$0 (top-$\theta^{(t)}$1 singular vector subspaces for $\theta^{(t)}$2) are computed; larger mean rotation angles correspond to greater OOD forgetting. RL “restores” OOD by partially realigning these subspaces, reducing the angular deviation induced by over-specialized SFT.

Ablation of $\theta^{(t)}$3 back to earlier SFT checkpoints can recover or erase OOD generalization, while manipulating $\theta^{(t)}$4 is nearly inert.

5. Key Takeaways, Best Practices, and Theoretical Implications

• SFT rapidly aligns (“hard-specializes”) LLM parameters to ID data/modalities, producing early OOD generalization, then over-specializes, rotating singular vectors away from robust OOD-supporting modes.
• RL, given a “recoverable” SFT checkpoint, acts to softly undo these rotations, restoring previously accessible OOD behavior—but is fundamentally limited by the best OOD that SFT originally achieved.
• Monitoring “rotation magnitude” via principal angle metrics offers a direct diagnostic of OOD risk beyond standard loss curves.
• The optimal SFT+RL protocol avoids both insufficient (underfit) and excessive (overspecialized) SFT, emphasizing an intermediate “sweet spot” for policy handoff.

Future work could include spectral-directional regularization during SFT to minimize harmful rotations, potentially obviating the need for RL-based restoration, and would likely further increase OOD maximality in a single-stage process.

6. Quantitative Benchmarks

Model	SFT Peak OOD	SFT End OOD	RL End OOD	ID End Acc
LLaMA-11B	18–19% (c.140)	~10%	16–18%	>80%
Qwen-7B	18–19% (c.120)	~10%	16–18%	>80%

RL endpoints match—never exceed—the SFT-peak OOD accuracy. OOD cannot be reliably inferred from ID metrics. Proper protocol selection is essential for robust generalization.

---

References:

• "RL Fine-Tuning Heals OOD Forgetting in SFT" (Jin et al., 8 Sep 2025)