---
title: Orthogonality of SFT and RL Updates in LLMs
url: https://www.emergentmind.com/topics/orthogonality-of-sft-and-rl-parameter-updates
type: topic
---

# Orthogonality of SFT and RL Updates in LLMs

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) parameter update orthogonality refers to the geometric relationship between these distinct optimization regimes in high-dimensional parameter space during the post-training of large language models (LLMs). This concept addresses whether—and under what conditions—updates from SFT and RL interfere, compete, or instead act independently (“orthogonally”) on different functional subspaces, with profound implications for catastrophic forgetting, knowledge consolidation, reasoning ability, and generalization.

## 1. Geometric Definition and Theoretical Context

Orthogonality of SFT and RL parameter updates formalizes whether the update vectors
\[
\Delta\theta_\text{SFT}\quad\text{and}\quad\Delta\theta_\text{RL}
\]
(where $\Delta\theta_\text{SFT}$ is the change from SFT and $\Delta\theta_\text{RL}$ from RL) satisfy $\langle \Delta\theta_\text{SFT},\,\Delta\theta_\text{RL} \rangle\approx 0$ (zero or near-zero cosine similarity) in the ambient parameter space. Strict orthogonality implies no destructive interference: one type of update does not undo or affect what the other achieves. In contrast, non-orthogonality (positive or negative cosine similarity) implies coupling; gains along one objective (e.g., cross-entropy minimization in SFT or reward maximization in RL) degrade the other, reflecting incompatible or antagonistic update directions.

This geometric view connects to functional specialization, subspace disentanglement, and error decomposition in high-dimensional models, offering a principled lens for understanding optimization interference and the limitations of naïve sequential or alternating fine-tuning.

## 2. Empirical Evidence from Parameter-Space Analysis

Direct evidence for SFT/RL orthogonality comes from explicit measurement of parameter update alignment on modern LLMs:

- **Cosine Similarity of Updates:** "Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation" quantifies the cosine similarity at the parameter and layer level after sequential SFT and RL. On Qwen2.5-7B-Instruct, all transformer layers yield cosines in $[-0.05, +0.05]$, confirming near-orthogonality. As a control, SFT-on-disjoint-data yields cosines $0.3$–$0.6$, demonstrating that this is not a generic artifact of high-dimensionality, but reflects genuine subspace separation [2601.11258]. Theoretical justification further shows, assuming LayerNorm-induced isotropy, negligible expected overlap of propagated signals between SFT and RL update directions.

- **Spectral Analysis:** In "RL Fine-Tuning Heals OOD Forgetting in SFT," SVD-based diagnostics on weight matrices measure rotation and cosine similarity in the subspaces spanned by the principal singular vectors (U, V). The cosine similarity between SFT-induced and RL-induced rotations is near zero for almost all layers and singular directions ($0.01\pm0.04$ in heads, slightly negative in intermediate indices), indicating that RL reverses or undoes SFT-induced rotations in largely orthogonal directions [2509.12235]. Analogously, "RL Is Neither a Panacea Nor a Mirage" and "The Path Not Taken" demonstrate through principal-subspace projectors and update alignment that RL fine-tuning changes low-curvature, off-principal directions orthogonal to the high-energy subspace dominantly affected by SFT [2508.16546, 2511.08567].

- **PRISM and Gradient Concentration:** While "Consolidation or Adaptation? PRISM" does not directly measure cosine similarity, it introduces a routing scheme based on gradient-concentration (Gini, kurtosis, coefficient of variation) to allocate data to SFT (diffuse, broad-gradient updates) or RL (concentrated, localized updates). The separation ensures that SFT updates do not interfere with RL-operated subspaces, achieving empirical near-orthogonality at the regime level, as evidenced by ablation studies and superior downstream task performance [2601.07224].

## 3. Counterevidence: Gradient Coupling and Theoretical Non-Orthogonality

Not all studies support strict orthogonality. "On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training" proves, with formal theorems and empirical validation, that SFT and RL updates are coupled: each loss gradient has a nonzero inner product with the other's gradient, in both SFT-then-RL or RL-then-SFT orderings. Specifically,
\[
\langle \nabla_\theta \mathcal{L}_\text{SFT},\, \nabla_\theta \mathcal{J}_\text{RL}\rangle \neq 0
\]
implying that improving one loss increases regret for the other. This coupling manifests in observed increases in SFT test loss upon RL, and reward degradation upon SFT after RL, on Qwen3-0.6B [2601.07389]. Mitigation involves mixture or constraint-based optimization, but complete decoupling via subspace orthogonality is theoretically unattainable whenever objectives differ.

Similarly, in the context of vision-language models (VLMs), "The Synergy Dilemma" observes that SFT and RL parameter updates are partially misaligned but not strictly orthogonal (cosine in $(0,1)$). All model-merging and interpolation attempts result in trade-offs, never full additive gains, consistent with "partial antagonism" in update directions [2507.07562].

## 4. Mechanistic Origins and SVD-Based Subspace Decomposition

A deeper mechanistic perspective arises from SVD analysis:

- **Principal Subspace vs. Orthogonal Complement:** In "The Path Not Taken," model parameters are decomposed into projections onto the top-k singular directions (principal subspace $P$) and their orthogonal complement $(I-P)$:
  \[
  \Delta W = P\Delta W + (I-P)\Delta W
  \]
  SFT updates align with $P$ (cosine similarity $\sim0.6$), produce spectral drift and subspace rotation, while RL updates target $(I-P)$ (cosine similarity $\sim0.1$), preserving spectrum and principal alignment.

- **Hessian Geometry:** The Three-Gate Theory posits that RL, due to KL constraints and local curvature, is forced into low-curvature, off-principal directions (flatter valleys in parameter space), while SFT exploits high-curvature, pre-learned modes for rapid pattern adjustment [2511.08567].

- **Rotation, Not Scaling:** Consistently, across several works, the singular-value spectrum is almost unchanged during SFT and RL; what shifts are the dominant singular vectors (i.e., the orientation of model capacities), with SFT causing large rotations that can drive OOD forgetting, and RL partially reversing or correcting these rotations—again in approximately orthogonal directions [2508.16546, 2509.12235].

## 5. Algorithms and Mitigation Strategies for Interference

Approaches to reduce destructive interference include:

- **Gradient Routing by Geometry:** "PRISM" routes data to SFT or RL based on the spatial geometry (concentration) of per-sample parameter gradients, ensuring that SFT produces broad, orthogonal tweaks, whereas RL reserves large updates for localized, conflict-inducing data [2601.07224].

- **Meta-Learned Fusion:** Bilevel optimization frameworks such as BRIDGE (in "Beyond Two-Stage Training") employ adaptive convex blending of SFT and RL gradients. While not enforcing hard orthogonality, this strategy reduces interference and adapts SFT signals to cooperate with RL by explicit meta-gradient updates of a cooperative gain term. Empirically, this yields better overall stability and efficiency [2509.06948].

- **Linear Skill Injection:** The near-orthogonality established in "Knowledge is Not Enough" enables modular transfer: an RL-derived skill vector can be linearly combined with a newly SFT-tuned model, directly injecting complex skills with minimal retraining, a strategy yielding strong empirical gains [2601.11258].

- **Spectrum-Aware Restoration:** Replacing or restoring top singular directions lost during SFT with their pre-fine-tuning values, or leveraging low-rank subspace merging, can rapidly recover generalization and OOD performance, sidestepping costly RL retraining [2508.16546, 2509.12235].

## 6. Practical Impact and Limitations

Empirical consequences of SFT/RL update geometry are robust:

- **Catastrophic Forgetting Mitigation:** Disentangling SFT and RL regimes, via gradient routing or subspace decoupling, improves final performance, reduces RL sample complexity, and prevents catastrophic forgetting of early SFT-acquired knowledge [2601.07224, 2509.12235].
- **Skill–Knowledge Modularization:** Modular skill transfer via orthogonal composition supports rapid adaptation to new domains or tasks without repeated, costly RL optimization [2601.11258].
- **Boundaries of Recovery:** When the SFT stage induces excessive singular-vector rotation (overfitting), even an orthogonal RL update cannot fully recover lost OOD capabilities, imposing practical constraints on sequential two-stage protocols [2508.16546].

However, strict orthogonality is neither universal nor unconditional. The existence of objective coupling, especially in domains or architectures where SFT and RL loss landscapes substantially overlap, precludes total independence. Some reported approaches may only achieve approximate or partial orthogonality, and intervening with hard constraints (e.g., projection of gradients) is not always practical or effective given the complex, evolving geometry of LLM parameter space [2601.07389, 2507.07562].

## 7. Summary Table: Empirical Findings on SFT/RL Update Orthogonality

| Paper                                   | Orthogonality Metric      | Result                             |
|------------------------------------------|---------------------------|-------------------------------------|
| "Knowledge is Not Enough" [2601.11258]   | Update cosine similarity  | Near-zero cosine: strong orthogonality (all layers) |
| "RL Fine-Tuning Heals OOD Forgetting" [2509.12235] | Singular-vector rotation & cosine | Cosine $\approx$ 0 in most regions; RL undoes SFT rotations orthogonally |
| "The Path Not Taken" [2511.08567]        | Principal subspace projection | SFT: align with principal; RL: align off-principal; overlap $\sim$ 0.08 |
| "PRISM" [2601.07224]                     | Gradient concentration   | Indirect evidence: regime separation reduces interference |
| "Non-decoupling..." [2601.07389]         | Gradient inner product   | Inner products nonzero: strict orthogonality provably absent |
| "Synergy Dilemma" [2507.07562]           | (Not directly measured)  | Partial misalignment (cos $\in$ (0,1)); trade-offs in interpolation curves |

## References

- "Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation" [2601.11258]
- "RL Fine-Tuning Heals OOD Forgetting in SFT" [2509.12235]
- "The Path Not Taken: RLVR Provably Learns Off the Principals" [2511.08567]
- "Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration" [2601.07224]
- "On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training" [2601.07389]
- "RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs" [2508.16546]
- "The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs" [2507.07562]
- "Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning" [2509.06948]

Source: https://www.emergentmind.com/topics/orthogonality-of-sft-and-rl-parameter-updates