Trust-Region Supervised Fine-Tuning

• TrSFT is a suite of methods that regularize supervised fine-tuning by enforcing a trust region to prevent policy drift and catastrophic forgetting.
• It employs techniques like gradient clipping, soft KL penalties, and noise-based regularization to stabilize model updates during mixed RL-SFT training.
• Variants such as TRAPO, PSFT, ASFT, Minor SFT, and R3F/R4F demonstrate improved performance and stability across diverse fine-tuning and cross-domain adaptation tasks.

Trust-Region Supervised Fine-Tuning (TrSFT) is a suite of methods for regularizing supervised fine-tuning (SFT) of LLMs and other neural networks by constraining policy or representation drift relative to an initial reference model. TrSFT methods introduce an explicit or implicit "trust region"—typically via per-example or per-token weighting, gradient clipping, soft KL penalties, or noise-based regularization—to prevent catastrophic forgetting, over-confident updates, or degenerate solution modes that arise when pure SFT is interleaved with reinforcement learning (RL) or cross-domain adaptation. Examples include the TrSFT component of TRAPO (Su et al., 19 Dec 2025), Proximal SFT (Zhu et al., 25 Aug 2025), Anchored SFT (Zhu et al., 28 Sep 2025), Minor SFT (Xie et al., 2024), and representational noise regularization (Aghajanyan et al., 2020).

1. Trust-Region Principle and Motivating Instabilities

Conventional SFT minimizes the forward KL divergence $\mathrm{KL}(P_E \,\|\, \pi_\theta)$ between an expert distribution $P_E$ and the model policy $\pi_\theta$. This "mode-covering" loss ensures mass everywhere the expert assigns nonzero probability, but can inflate probability in unsupported "void" regions and destabilize subsequent RL stages (Su et al., 19 Dec 2025). Policy-drift and representational collapse are observed: SFT unconstrainedly fits new data, eroding prior capabilities and causing oscillations or entropy collapse (Zhu et al., 25 Aug 2025, Aghajanyan et al., 2020).

Trust-region SFT (TrSFT) restricts updates within a bounded divergence to a reference policy or representation, mitigating these risks. The trust region can be enforced strictly (via hard KL/divergence constraints), softly (via clipped gradients, surrogate losses, or parametric noise), or adaptively (as in micro-group sampling). This design stabilizes fine-tuning and supports interleaving with RL or out-of-domain generalization.

2. Mathematical Formulations in TrSFT

Approaches operationalize the trust region via diverse mechanisms, each constraining drift differently:

• TrSFT (TRAPO) (Su et al., 19 Dec 2025): For each expert prefix token $y_i$, the SFT gradient $-\partial\,\log\,\pi_\theta(y_i|x,y_{<i})$ is weighted:

$$w_{\rm TrSFT}(p) = \begin{cases} \frac{1}{p} & p>\alpha \ \frac{1}{\alpha} & p\le\alpha \end{cases}$$

where $p=\pi_\theta(y_i|x,y_{<i})$ and $\alpha$ is a fixed trust-region threshold. Inside the trust region ($p>\alpha$), standard forward-KL SFT is applied; outside, gradient weights are clipped, preventing large updates on low-confidence tokens. The mode-seeking endpoint corresponds to pruned expert modes for $P_E(c)\le\alpha$.

• Proximal SFT (PSFT) (Zhu et al., 25 Aug 2025): Inspired by PPO, imposes a clipped importance-weighted surrogate for every token:

$$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\rm old}}(a_t|s_t)}$$

and loss

$$L^{\mathrm{PSFT}}(\theta) = -\mathbb{E}[\min(r_t, \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon))]$$

This enforces a soft local trust-region by bounding each token's probability update by $\epsilon$.

• Anchored SFT (ASFT) (Zhu et al., 28 Sep 2025): Adds a reverse KL penalty to DFT-reweighted SFT:

$$L_{\rm ASFT}(\theta) = L_{\rm DFT}(\theta) + \lambda \mathbb{E}_{s\sim D}[\mathrm{KL}(\pi_\theta(\cdot|s)\|\pi_{\rm base}(\cdot|s))]$$

Here, $\lambda$ controls the trust-region size, pulling the current policy distribution toward the frozen base model.

• Minor SFT (Xie et al., 2024): Applies a sample-wise sigmoid weighting based on the log-probability ratio to the reference:

$$w = 2 \cdot \sigma(-\beta\,\Delta(x,y)),\qquad \Delta(x,y) = \log \frac{\pi_\theta(y|x)}{\pi_{\theta_0}(y|x)}$$

and loss

$$L_{\rm MinorSFT} = -\mathbb{E}\left[\frac{w}{m} \sum_{t=1}^m \log \pi_\theta(y_t|x,y_{<t})\right]$$

Strong deviation is suppressed by weighting, implementing a soft constraint.

• R3F/R4F Representational Trust Region (Aghajanyan et al., 2020): Regularizes by adding small parametric noise to the encoder and penalizing symmetric KL between outputs:

$$L_{\rm R3F}(\theta) = L_{\rm SFT}(\theta) + \lambda\,\mathrm{KL}_S[ p, p']$$

$p$ = model head output, $p'$ = output with noise-perturbed encoder activations.

These methods are summarized in the table below:

Method	Trust-Region Mechanism	Constraint Reference
TrSFT (TRAPO)	Gradient clipping by $1/\alpha$	Policy $\pi_\theta$ itself
PSFT	PPO-style clipped ratio	Previous iteration policy
ASFT	KL penalty (reverse KL)	Fixed base model
Minor SFT	Sigmoid downweighting	Fixed reference model
R3F/R4F	Symmetric KL on perturbed outputs	Clean vs. perturbed representations

3. Optimization Workflows and Pseudocode

Concrete optimization workflows implement the trust region through per-batch or per-token manipulations:

• TrSFT (Su et al., 19 Dec 2025):
  1. For each batch, sample expert prefixes.
  2. Apply per-token trust-region-weighted SFT loss.
  3. For continuation tokens, interleave with RL losses (e.g. GRPO).
  4. Update model on sum of SFT and RL objectives.
  5. Trust-region parameter $\alpha$ is fixed.

• PSFT (Zhu et al., 25 Aug 2025):

  • After optional SFT warm-up, for each batch:
  • 1. Compute ratios $r_t$ for each token.
  • 2. Clip $r_t$ in $[1-\epsilon, 1+\epsilon]$.
  • 3. Minimize negative of the minimum of $r_t$, $\mathrm{clip}(r_t)$ over batch.
• ASFT (Zhu et al., 28 Sep 2025):
  • For each minibatch:
  • 1. Compute model probabilities on ground-truth.
  • 2. Weight SFT loss by (stop-gradient) model probability.
  • 3. Add reverse KL penalty to base model distribution.
  • 4. Update using AdamW.
• Minor SFT (Xie et al., 2024):
  • For each batch,
  • 1. Compute per-example log-probability ratio vs. reference.
  • 2. Weight cross-entropy loss per token by $2\,\sigma(-\beta\Delta)$.
  • 3. Monitor average deviation.
• R3F/R4F (Aghajanyan et al., 2020):
  • Add symmetric KL between clean and noise-perturbed head predictions as a regularizer per instance (with or without spectral normalization of the head).

All pseudocode implementations emphasize in-batch calculation of reference probabilities and per-example constraint application.

4. Theoretical Insights and Stability Properties

The trust-region mechanism alleviates several theoretical and empirical pathologies:

• Distribution Blending in SFT: Standard SFT gradients with unbounded $1/p$ can force the model to assign probability in unsupported output regions, harming later RL. TrSFT's gradient clipping prevents this (Su et al., 19 Dec 2025).
• Mode Seeking vs. Mode Covering: TrSFT and ASFT interpolate between forward-KL (mode-covering) and reverse-KL (mode-seeking), stabilizing training. The pruning at the optimal solution of TrSFT is explicit: low-probability expert modes ($P_E(c)\le\alpha$) are zeroed out, with the model concentrating on dominant modes (Su et al., 19 Dec 2025).
• Entropy Dynamics: PSFT demonstrates smooth entropy trajectories and avoids collapse, unlike standard SFT which shows sawtooth entropy drops associated with overfitting. This protects both in-domain and out-of-domain performance (Zhu et al., 25 Aug 2025).
• Representational Collapse: R3F/R4F regularization maintains encoder representations closer to pre-training and yields higher probing accuracies even after repeated fine-tuning cycles (Aghajanyan et al., 2020).
• Sample-wise Early Shutoff: Minor SFT's per-example weight decays suppress learning on examples once the model's likelihood overshoots the reference, implicitly enforcing a soft KL bound (Xie et al., 2024).

5. Practical Implementations and Hyperparameters

Recommended trust-region hyperparameters reflect a trade-off between stability and learning progress:

• TrSFT (TRAPO): $\alpha=0.1$, batch size $128$, up to $8192$ context, RL learning rate $5\times 10^{-6}$, SFT $5\times 10^{-5}$ (Su et al., 19 Dec 2025).
• PSFT: Clipping threshold $\epsilon\approx 0.2-0.28$, learning rate $1\times 10^{-6}$, batch size $256$ (Zhu et al., 25 Aug 2025).
• ASFT: KL penalty $\lambda\in[0.05,0.1]$, batch size $32-256$, learning rate $5\times 10^{-6}$ to $2\times 10^{-4}$ (Zhu et al., 28 Sep 2025).
• Minor SFT: Penalty strength $\beta\approx 0.04$, learning rate $2\times 10^{-5}$. Grid search over $(\mathrm{lr},\beta)$ is recommended (Xie et al., 2024).
• R3F/R4F: Noise scale $\sigma\sim 1\times 10^{-5}$, regularization $\lambda=0.1$ to $5.0$, standard Adam; spectral norm if R4F (Aghajanyan et al., 2020).

Design best practices include freezing a reference base model, in-batch reference calculation, and monitoring of divergence metrics to ensure bounded update drift and prevent collapse.

6. Empirical Results and Benchmarks

Extensive experiments confirm that TrSFT methods deliver consistent improvements in downstream accuracy, stability, and generalization across domains:

• TRAPO (Qwen2.5-Math-7B) (Su et al., 19 Dec 2025):
  • Math-average accuracy: $56.6\%$ (vs. $50.3\%$ SFT, $50.4\%$ RL, $54.3\%$ SFT$\to$RL, $53$–$55.5\%$ SFT+RL baselines)
  • General benchmarks: $68.3\%$ (vs. next best $66.7\%$)
  • Micro-group sampling alone yields $+2.3$ pts over RL; standard SFT degrades performance by $-18$ pts; TrSFT restores stability with $+3.9$ pts over micro-group+RL.
• PSFT (Zhu et al., 25 Aug 2025):
  • In-domain: matches SFT (e.g., Qwen2.5-7B $46.98$ vs. $47.99$).
  • Out-of-domain: improves generalization (Qwen2.5-7B $61.26$ vs. SFT $57.90$).
  • Maintains stable entropy and returns under prolonged training.
• ASFT (Zhu et al., 28 Sep 2025):
  • Medical: $42.03\%$ (vs. $33.37\%$ SFT, $29.19\%$ DFT), math reasoning: $28.75\%$ (vs. $16.73\%$ SFT).
  • Code generation (HumanEval, MBPP): $27.0\%$ (vs. $26.4\%$ SFT).
  • Maintains low KL divergence to base and avoids collapse.
• Minor SFT (Xie et al., 2024):
  • Higher accuracy than SFT and SFT using DPO on FinanceIQ, FineEval, C-Eval.
  • Reduced deviation metric throughout training.
• R3F/R4F (Aghajanyan et al., 2020):
  • Improved GLUE and XNLI performance, higher representational probing accuracy, and lower computational cost compared to previous adversarial methods.

7. Extensions, Variants, and Comparative Perspectives

Variants of TrSFT leverage either explicit divergence penalties or implicit sample-wise weighting, and can be integrated with RL (GRPO, PPO) or preference-based alignment. Some key distinctions:

• Explicit KL (ASFT, PSFT): Direct control over the trust-region radius; clear analog with TRPO and PPO in RL (Zhu et al., 25 Aug 2025, Zhu et al., 28 Sep 2025).
• Clipping/Sigmoid weighting (TrSFT/Minor SFT): Implicit, automatic early shutoff on high-deviation examples; minimal parameterization (Su et al., 19 Dec 2025, Xie et al., 2024).
• Representation-level (R3F/R4F): Focus on encoder drift rather than output probability; particularly relevant for transfer and multi-task generalization (Aghajanyan et al., 2020).
• Multi-method frameworks (TRAPO): TrSFT is embedded within an RL-interleaved framework, stabilized via trust-region SFT and adaptive micro-group prefixing (Su et al., 19 Dec 2025).

All TrSFT approaches serve to regularize SFT, protect pre-trained capabilities, and balance imitation with exploration, especially when SFT and RL are dynamically mixed. Empirical ablations consistently show that trust-region enhanced SFT prevents mode collapse, reduces drift, and yields higher stable downstream performance than unregularized SFT or naively interleaved SFT+RL pipelines.