Adaptive Supervised Fine-Tuning

• Adaptive Supervised Fine-Tuning (Adaptive SFT) is a paradigm that dynamically selects task-relevant parameters, data, and objectives to efficiently fine-tune large language models.
• It incorporates techniques such as Core Parameter Isolation, selective data curation, and dynamic task grouping to mitigate catastrophic forgetting and optimize compute utilization.
• Empirical results demonstrate significant improvements, including reduced training time by up to 45% and marked enhancements in multi-task and low-resource performance.

Adaptive Supervised Fine-Tuning (Adaptive SFT) encompasses a class of algorithms and frameworks designed to overcome inefficiencies and negative transfer effects endemic to conventional supervised fine-tuning of LLMs, especially in multi-task, domain adaptation, and resource-constrained settings. Unlike traditional SFT, which uniformly updates all model parameters on all data, Adaptive SFT incorporates explicit parameter, data, or objective selection mechanisms—often dynamically or in a task-aware fashion—to mitigate catastrophic forgetting, task interference, and wasted compute. Recent instantiations include Core Parameter Isolation Fine-Tuning (CPI-FT), selective data curation (SLearnLLM), curriculum-informed task grouping, and meta-learned loss balancing.

1. Motivation and Challenges in Standard SFT

Conventional supervised fine-tuning of LLMs often suffers from the “seesaw phenomenon”: improvements on some tasks (or domains) come at the expense of declines in others due to indiscriminate parameter updates and destructive interference (Wang et al., 29 Aug 2025). In multi-task settings, updating the full parameter set for all tasks can degrade performance on previously learned tasks (catastrophic forgetting), while also leading to inefficient compute and suboptimal knowledge transfer. In domain adaptation, redundancy between the SFT dataset and model’s existing knowledge yields marginal gains but high training costs (Liu et al., 23 May 2025). These issues motivate the need for Adaptive SFT paradigms that can:

• Isolate and update only task-relevant (“core”) parameters.
• Intelligently curate or filter training data for informative or “unknown” knowledge.
• Adaptively balance or compose task-specific training objectives.
• Dynamically group related tasks for joint modelling, based on deep parameter or feature correspondences.
• Integrate SFT with reinforcement learning (RL) schemes while controlling trade-offs between imitation and exploration.

2. Core Parameter Isolation Methods

A key line of work formalizes Adaptive SFT as the identification and isolation of “core” parameter regions per task. The Core Parameter Isolation Fine-Tuning (CPI-FT) framework (Wang et al., 29 Aug 2025) exemplifies this approach, introducing the following workflow steps:

1. Core Region Discovery: For each task $T_i$, CPI-FT performs a short SFT probe, computes per-parameter update magnitudes $$\Delta|\theta^{(i)}_j|=|\theta^{(i)}_j-\theta^{(0)}_j|$$, and defines the “core” $C_i$ as the top $p\%$ of largest-magnitude parameters.
2. Task Grouping: Pairwise Jaccard overlap $S(C_i, C_j)$ is computed between all task cores; tasks with $S\geq\tau$ are clustered for joint modelling.
3. Parameter Fusion: For each group, core regions are transplanted directly (overwritten) from single-task SFT; non-core parameters are fused using Spherical Linear Interpolation (SLERP), based on angular distance between parameter vectors.
4. Pipelined SFT with Masking: Subsequent mixed-task fine-tuning freezes the union of previously found core regions, updating only unfrozen parameters. The process proceeds groupwise.
5. Empirical Results: Across LLaMA-2-7B and other models, CPI-FT outperforms vanilla multitask SFT by up to +4.9 points (AvgNorm), sharply reduces catastrophic forgetting (−24.5 reduced to −5.7 on two-task transfer), and better preserves accuracy on low-resource tasks under data imbalance (Wang et al., 29 Aug 2025).

Approach	Forgetting (ΔA)	Low-resource Task Gain	AvgNorm (LLaMA-2-7B, $p=1\%$, $\tau=0.1$)
Multi-task SFT	−24.5	baseline	6.58
CPI-FT	−5.7	+3–4 points	7.21

3. Selective and Self-Filtering Data Strategies

Adaptive SFT also encompasses approaches that reduce compute and maximize statistical efficiency via intelligent data selection. The SLearnLLM framework (Liu et al., 23 May 2025) utilizes the model’s own self-assessment to filter SFT datasets:

• Self-grading: The pretrained model predicts answers for each data point $(x_i, y_i)$, then uses its own (or another LLM’s) chain-of-thought-based critic to grade correctness.
• Filtering: Only examples judged incorrect (where the model’s current knowledge is deficient) are included in the fine-tuning set $D_\mathrm{filtered}$.
• Efficiency: Empirically, more than 40–60% training cost is saved with <0.5pp loss in accuracy, across agriculture and medicine datasets, and up to 44% compute savings on 32B models (Liu et al., 23 May 2025).

This strategy ensures that computation is focused on “unknown knowledge,” rather than expending resources on redundancy.

4. Adaptive Task Grouping and Curriculum

Adaptive SFT frequently incorporates task grouping and dynamic curricula based on deep correspondences in parameter or attention patterns. In CPI-FT (Wang et al., 29 Aug 2025), task grouping is based on core-region overlap; tasks with high Jaccard similarity in their update patterns are jointly trained to exploit transfer and minimize interference.

Similarly, works leveraging attention-head activation patterns have shown that complex tasks' adaptation often follows compositional rules over basic subtasks, allowing for targeted two-phase curriculum design: first fine-tune on a linear mix (by activation drift) of basic task data, then on scarce complex task data, achieving superior adaptation efficiency compared to random mixing or standard SFT (Zhao et al., 24 Sep 2024).

5. Dynamic and Meta-Learned Integration with Reinforcement Learning

Several frameworks propose adaptively integrating SFT and RL, leveraging their complementary strengths for reasoning and alignment. Notable paradigms include:

• Step-wise Adaptive Switching (SASR): Dynamically interpolates SFT and RL updates based on the gradient norm of the SFT loss and the KL divergence to the data distribution. When the model is “far” (high gradient norm), SFT dominates; as the gap closes, RL steps become more frequent. This approach unifies objectives and ensures smooth curriculum adaptation without catastrophic forgetting. Empirically, SASR achieves superior accuracy across GSM8K, MATH, and logic datasets compared to static and hybrid schedules (Chen et al., 19 May 2025).
• Meta-Learned SFT–RL Balancing (AMFT): Treats the balance parameter $\mu$ (weight between SFT’s implicit reward and RL's explicit reward) as a learnable meta-parameter. A meta-gradient controller, regularized by policy entropy, dynamically tunes $\mu$ to optimize long-term generalization. AMFT achieves new state-of-the-art ID/OOD accuracy on mathematical reasoning and vision-language navigation, and ablation studies confirm the indispensability of both meta-gradient and entropy heuristics (He et al., 9 Aug 2025).
• Online Interleaving (ReLIFT): SFT is selectively invoked only on those instances where RL fails completely (reward=0), using demonstration data for the hardest questions. RL and SFT steps are interleaved, not blended in loss, yielding strong generalization, robust hard-question accuracy, and high sample efficiency (Ma et al., 9 Jun 2025).

6. Adaptive Optimization Objectives and Parameter Fusion

Adaptive SFT also involves generalizations of the loss function and parameter update strategy:

• Group-Wise and Token-Adaptive Objectives: SFT-GO partitions tokens in each sequence by importance and jointly optimizes cross-entropy and the “worst-group” loss. Grouping can be guided by TF–IDF, semantic compression, or loss-based methods. Theoretical analysis guarantees the worst-group loss cannot increase relative to standard SFT, and empirical metrics show consistent +1–2 point gains across multiple benchmarks (Kim et al., 17 Jun 2025). Token-Adaptive Loss Reweighting (TALR) down-weights high-loss (“hard”) tokens in order to minimize general capability loss during aggressive domain SFT (Lin et al., 25 Sep 2025).
• Parameter Fusion via SLERP: In CPI-FT's pipeline, fused parameter vectors are constructed by directly overwriting core regions and geometrically interpolating the remainder via SLERP. This avoids catastrophic parameter conflict between tasks (Wang et al., 29 Aug 2025).

7. Empirical Performance and Practical Guidelines

Across frameworks, Adaptive SFT consistently outperforms standard SFT schemes in benchmarks involving multitask, domain transfer, low-resource, or reasoning-intensive regimes. Principal findings:

• CPI-FT: +14–34% reduction in catastrophic forgetting and up to +5.1% Avg. Norm. improved task average (Wang et al., 29 Aug 2025).
• SLearnLLM: 30–45% training time reduction with <0.5pp validation gap (Liu et al., 23 May 2025).
• SASR/AMFT: AMFT achieves 61.3% ID and 63.3% OOD accuracy in math reasoning, exceeding all static SFT→RL baselines (He et al., 9 Aug 2025). SASR demonstrates up to +2–5pp accuracy increase over standard and hybrid methods (Chen et al., 19 May 2025).
• TALR: Up to 5pp absolute increase in general benchmark scores for a fixed domain accuracy, with negligible training overhead (Lin et al., 25 Sep 2025).

General recommendations converge on monitoring gradient/KL metrics for curriculum adaptation, the use of light-weight data/self-knowledge filters, and the integration of information-theoretic or meta-learning controllers for objective blending. Parameter isolation and attention-pattern analyses offer additional pathways for highly efficient, robust, and task-adaptive SFT deployment.