Rotation-Preserving Supervised Fine-Tuning (RPSFT)

• RPSFT is a supervised fine-tuning method that penalizes shifts in pretrained dominant singular subspaces to maintain general-purpose capabilities.
• It leverages SVD to isolate the top-k singular block, controlling rotations and mitigating degradation in out-of-domain performance.
• Empirical results demonstrate that RPSFT balances in-domain improvements with OOD retention, achieving superior performance metrics.

Searching arXiv for the specified papers and related RPSFT context. {"query":"(Jin et al., 8 May 2026) Rotation-Preserving Supervised Fine-Tuning", "max_results": 5} {"query":"(Ma et al., 2024) Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation", "max_results": 5} {"query":"RL Fine-Tuning Heals OOD Forgetting in SFT arXiv (Jin et al., 8 Sep 2025)", "max_results": 5} Rotation-Preserving Supervised Fine-Tuning (RPSFT) is a modification of standard supervised fine-tuning for LLMs that explicitly constrains how much the dominant pretrained singular directions of each weight matrix are allowed to move. It penalizes changes in the projected top-$k$ singular-vector block of each pretrained weight matrix, limiting unnecessary rotation while preserving task adaptation. The method is motivated by the observation that supervised fine-tuning improves in-domain performance but can degrade out-of-domain generalization, and by evidence that this degradation is related to rotations of dominant singular subspaces rather than large changes in singular values (Jin et al., 8 May 2026, Jin et al., 8 Sep 2025).

1. Problem formulation and research context

Given a pretrained model with parameters $\theta_0$ and a supervised dataset $\mathcal{D}=\{(x,y)\}$, standard SFT minimizes the negative log-likelihood

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$

starting from $\theta=\theta_0$. In LLMs, this is the usual instruction tuning or domain-specific post-training. The central problem addressed by RPSFT is the stability–plasticity trade-off: full-parameter SFT is plastic enough to improve in-domain performance, but often insufficiently stable to preserve broader pretrained capabilities (Jin et al., 8 May 2026).

A crucial empirical precursor came from the study of two-stage full-parameter SFT followed by full-parameter RL on a controlled arithmetic reasoning game. That work found that OOD performance peaks at the early stage of SFT and then declines; the best SFT checkpoint cannot be captured by training/test loss; the subsequent RL stage plays an OOD restoration role; and the OOD behavior strongly correlates with the rotation of singular vectors, while singular values are actually quite stable throughout fine-tuning (Jin et al., 8 Sep 2025). The implication is not merely that SFT changes parameters, but that it changes them geometrically: continued SFT can rotate dominant directions toward in-domain specialization and away from OOD-supporting subspaces.

RPSFT responds to that picture by making rotation control an explicit part of the SFT objective. Rather than attempting to compute Hessian or Fisher eigenspaces directly at LLM scale, it uses dominant pretrained singular directions as an efficient proxy for loss-sensitive directions, and regularizes motion within that proxy subspace (Jin et al., 8 May 2026).

2. Spectral geometry and the notion of preserved rotation

For each selected pretrained weight matrix $W^0 \in \mathbb{R}^{m\times n}$, RPSFT uses the singular value decomposition

$W^0 = U^0 \Sigma^0 (V^0)^\top,$

with top-$k$ singular vectors $U_0^{(k)} \in \mathbb{R}^{m\times k}$ and $V_0^{(k)}\in \mathbb{R}^{n\times k}$. The key object is the projected strict top block

$\theta_0$0

together with the pretrained reference block

$\theta_0$1

Because the basis comes from the SVD of $\theta_0$2, $\theta_0$3 is the leading $\theta_0$4 diagonal block of $\theta_0$5 (Jin et al., 8 May 2026).

This construction yields a selective notion of preservation. A global $\theta_0$6 anchor such as $\theta_0$7 constrains all coordinates equally. RPSFT instead anchors only how the current matrix acts on the dominant pretrained singular subspace: $\theta_0$8 The complementary directions remain free to adapt. In the terminology of the paper, RPSFT preserves projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions (Jin et al., 8 May 2026).

The motivation for using SVD rather than Fisher is empirical as well as computational. For a representative layer, the paper measures the fraction of total gradient energy captured by the strict top-$\theta_0$9 singular block in the pretrained SVD basis: $\mathcal{D}=\{(x,y)\}$0 For Llama-8B, Qwen-7B, and Qwen-3B, only $\mathcal{D}=\{(x,y)\}$1 of singular directions (top 768) already capture $\mathcal{D}=\{(x,y)\}$2 of this Fisher-projected energy (Jin et al., 8 May 2026). This supports the claim that dominant pretrained singular directions overlap strongly with loss- or Fisher-sensitive directions, while avoiding the intractability of directly manipulating full Hessian or Fisher structure.

3. Objective function, theoretical properties, and optimization

For a collection $\mathcal{D}=\{(x,y)\}$3 of selected matrices $\mathcal{D}=\{(x,y)\}$4, the RPSFT penalty is

$\mathcal{D}=\{(x,y)\}$5

The full training objective is

$\mathcal{D}=\{(x,y)\}$6

with protected rank $\mathcal{D}=\{(x,y)\}$7 and regularization strength $\mathcal{D}=\{(x,y)\}$8 (Jin et al., 8 May 2026).

Two boundary cases are central to the method’s interpretation. When $\mathcal{D}=\{(x,y)\}$9, the penalty vanishes and RPSFT reduces to vanilla SFT. When $$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$0, the penalty reduces to full weight-space $$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$1 anchoring, denoted “L2 Init” in the experiments (Jin et al., 8 May 2026). This places RPSFT between unconstrained SFT and global weight anchoring: it is neither pure plasticity nor uniform rigidity.

The paper also gives a blockwise interpretation. In an extended orthogonal basis $$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$2,

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$3

and the RPSFT penalty is simply

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$4

Only the top-left block is regularized; $$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$5, $$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$6, and $$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$7 are unconstrained (Jin et al., 8 May 2026). This formalizes the method’s selective geometry: adaptation is permitted, but not arbitrary drift in the dominant pretrained subspace.

At a stationary point of

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$8

the paper derives

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-\log \pi_\theta(y\mid x)\big],$$9

together with the drift bound

$\theta=\theta_0$0

Under continuous-time gradient flow, the protected coordinates behave as an exponential damping filter on task-induced drift within the dominant subspace (Jin et al., 8 May 2026). This suggests that the regularizer does not eliminate task learning; rather, it attenuates exactly the component of learning that most strongly perturbs the protected spectral block.

Algorithmically, RPSFT precomputes truncated SVDs for each selected pretrained matrix, caches $\theta=\theta_0$1, $\theta=\theta_0$2, and $\theta=\theta_0$3, and then adds the projected-block penalty during each minibatch update. The paper applies the method to all major linear matrices in transformer blocks: attention $\theta=\theta_0$4, $\theta=\theta_0$5, $\theta=\theta_0$6, $\theta=\theta_0$7, and MLP $\theta=\theta_0$8, $\theta=\theta_0$9, $W^0 \in \mathbb{R}^{m\times n}$0. Empirically, RPSFT adds $W^0 \in \mathbb{R}^{m\times n}$1 GPU-hours relative to vanilla SFT, and peak memory is below IW and L2 Init (Jin et al., 8 May 2026).

4. Relation to other rotation-preserving fine-tuning paradigms

RPSFT belongs to a broader geometric family of methods that attempt to preserve pretrained structure during adaptation. A prominent earlier example is Orthogonal Fine-Tuning and its Givens-based extension. In that line, a linear layer with pretrained weights $W^0 \in \mathbb{R}^{m\times n}$2 is adapted by left-multiplying an orthogonal matrix $W^0 \in \mathbb{R}^{m\times n}$3,

$W^0 \in \mathbb{R}^{m\times n}$4

so that inner products, norms, angles, and cosine similarities between weight vectors are preserved under exact rotation. The 2024 qGOFT paper replaces the $W^0 \in \mathbb{R}^{m\times n}$5 Cayley-style parameterization with a chain of Givens rotations, achieving $W^0 \in \mathbb{R}^{m\times n}$6 parameters per layer, and then relaxes strict orthogonality with soft regularization in order to allow controlled norm and relative angular adjustments (Ma et al., 2024).

The difference from RPSFT is architectural and conceptual. qGOFT is a parameter-efficient reparameterization-based PEFT method: the base weights are frozen, and adaptation is realized through structured multiplicative transformations. RPSFT is full-parameter SFT with a spectral regularizer on selected pretrained subspaces. qGOFT therefore preserves geometry by constraining the form of the update itself, whereas RPSFT preserves geometry by penalizing drift in a pretrained SVD basis (Ma et al., 2024, Jin et al., 8 May 2026).

The 2025 SFT-to-RL study provides the bridge between these views. It showed that singular value reversion has almost no effect on performance, while singular vector reversion has dramatic effects on both ID and OOD accuracy, and that principal angles between base and later checkpoints can sweep from about $W^0 \in \mathbb{R}^{m\times n}$7–$W^0 \in \mathbb{R}^{m\times n}$8 degrees up to about $W^0 \in \mathbb{R}^{m\times n}$9 degrees (Jin et al., 8 Sep 2025). A plausible implication is that “rotation-preserving” names two related but technically distinct strategies: one constrains weight updates to be orthogonal or quasi-orthogonal transformations, and the other constrains the movement of pretrained dominant singular subspaces during full-parameter optimization.

5. Empirical performance and downstream RL behavior

The main RPSFT experiments use three instruction-tuned base models—Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Qwen2.5-3B-Instruct—fine-tuned on OpenR1-Math and then, in a second stage, used as initializations for DAPO RL on DAPO-Math-17k. In-domain evaluation covers AIME24, AIME25, AMC23, MATH-500, Minerva Math, and OlympiadBench; OOD evaluation covers GPQA, SuperGPQA, MMLU-Pro, IFEval, Safety Benchmark, and TruthfulQA (Jin et al., 8 May 2026).

On the SFT stage, RPSFT achieves the best or tied-best tuned ID averages and Pass@k across all three models. For Llama-8B, it improves Avg@k by $W^0 = U^0 \Sigma^0 (V^0)^\top,$0 and Pass@k by $W^0 = U^0 \Sigma^0 (V^0)^\top,$1 percentage points over base. On OOD metrics, Llama-8B shows $W^0 = U^0 \Sigma^0 (V^0)^\top,$2 pp on OOD Avg@k, and OOD Pass@k is nearly preserved with a slight net $W^0 = U^0 \Sigma^0 (V^0)^\top,$3 pp. For Qwen-7B and Qwen-3B, all SFT methods incur some OOD Avg@k degradation, but RPSFT reduces this degradation relative to vanilla SFT; for OOD Pass@k, RPSFT slightly improves over base for all three models (Jin et al., 8 May 2026).

The method’s geometric effects are visible beyond summary scores. Layerwise analyses show that RPSFT consistently reduces dominant-subspace rotation across $W^0 = U^0 \Sigma^0 (V^0)^\top,$4 and MLP projections. Hidden-state drift analysis finds that RPSFT centroids are closest to the base model in most panels, whereas SFT, IW, and especially DFT drift farther in representation space. These observations are consistent with the intended mechanism: better preservation of pretrained representations without eliminating in-domain adaptation (Jin et al., 8 May 2026).

The RL stage sharpens the same pattern. With DAPO initialization from SFT checkpoints, Llama-8B and Qwen-7B achieve the best final ID Avg@k and Pass@k across math benchmarks under RPSFT; Qwen-3B achieves the best final Avg@k, while IW slightly edges out Pass@k by approximately $W^0 = U^0 \Sigma^0 (V^0)^\top,$5 points. Across all three models, RPSFT yields the best final OOD Avg@k and Pass@k summary averages (Jin et al., 8 May 2026). This aligns with the earlier claim that RL often restores OOD capability rather than creating fundamentally new capability: a better-preserved SFT checkpoint provides a better starting point for RL than an over-rotated one (Jin et al., 8 Sep 2025).

6. Hyperparameters, limitations, and future directions

Rank selection and regularization strength are central. The paper’s practical rule is to choose the smallest $W^0 = U^0 \Sigma^0 (V^0)^\top,$6 such that the strict top block captures about $W^0 = U^0 \Sigma^0 (V^0)^\top,$7 of total gradient energy on a representative layer; in the reported 8B-scale experiments, this corresponds to $W^0 = U^0 \Sigma^0 (V^0)^\top,$8, approximately $W^0 = U^0 \Sigma^0 (V^0)^\top,$9 of rank. The default $k$0 works well, and rank sweeps show robustness across $k$1–$k$2, with degradation as the method approaches full-rank L2 Init (Jin et al., 8 May 2026). This suggests that moderate protection is preferable to either no protection or global anchoring.

Several limitations are explicit. The implementation uses the same protected rank $k$3 for all chosen matrices, even though different layers likely have different degrees of Fisher–SVD overlap. The method requires truncated SVD for each regularized matrix at initialization, which may become slow for very large models unless efficient approximate SVD or layer subsampling is used. Large $k$4 or large $k$5 can overly constrain the model and underfit ID tasks. The experiments are confined to 3B–8B Llama/Qwen models with math-focused training, so transfer to larger scales, multimodal settings, and non-math domains remains open (Jin et al., 8 May 2026).

The broader research trajectory also remains unsettled. The 2025 analysis identifies rotation of singular vectors as the key mechanism behind OOD forgetting and RL’s restorative effect, but it does not fully pin down why specific angles correspond to specific performance changes or how gradient flow on individual examples maps to singular vector rotations (Jin et al., 8 Sep 2025). The 2024 qGOFT results, meanwhile, indicate that explicit orthogonality or quasi-orthogonality can be effective in PEFT regimes, but they operate under a different optimization and capacity budget than full-parameter SFT (Ma et al., 2024). A plausible implication is that future work may converge on hybrid geometry-aware schemes: layer-wise adaptive $k$6 and $k$7, selective protection of early attention or other OOD-critical blocks, and combinations of spectral preservation with PEFT or RL.

In its current form, RPSFT is best understood as a geometrically informed regularizer for SFT. It does not freeze pretrained weights, nor does it enforce full orthogonality. Instead, it stabilizes the dominant pretrained directions that appear to encode high-curvature, general-purpose behavior, while leaving the rest of parameter space available for domain adaptation. Within that design space, it offers a concrete and computationally tractable answer to the problem of preserving OOD capability during supervised post-training (Jin et al., 8 May 2026).