Continuous Supervised Fine-Tuning (CSFT)

• Continuous Supervised Fine-Tuning (CSFT) is a methodology for incrementally adapting pre-trained neural networks to new supervised tasks while mitigating catastrophic forgetting.
• It leverages strategies such as non-parametric feature transformation, synthetic data replay, and dynamic parameter freezing to maintain past performance.
• CSFT is applied in on-device AI, specialized chat systems, and vision models, addressing operational constraints like data privacy and limited memory.

Continuous Supervised Fine-Tuning (CSFT) is a set of methodologies and algorithmic frameworks for incrementally adapting deep neural networks to new, supervised tasks or domains over time, while preserving prior knowledge and minimizing catastrophic forgetting. CSFT is broadly characterized by sequential fine-tuning of a pre-trained or previously fine-tuned model on new task data without full retraining, and usually under operational constraints such as data privacy, memory limitation, or lack of access to historical data. This survey synthesizes the algorithmic foundations, implementation strategies, and comparative performance of CSFT across vision and language domains.

1. Formalization and Motivations

In the canonical CSFT setting, model parameters $\theta_0$ are first initialized via pre-training or instruction tuning on a distribution $D_{\text{gen}}$. When new supervised data $D_{\text{dom}}$ arrives (representing a domain or task shift), standard fine-tuning solves

$$\min_\theta L_{\text{dom}}(\theta) = \mathbb{E}_{(x,y)\sim D_{\text{dom}}}\left[-\log P_\theta(y|x)\right]$$

yielding updated weights $\theta^*$. Catastrophic forgetting manifests when evaluation loss or accuracy on $D_{\text{gen}}$ (or previously encountered tasks) substantially degrades after the update: $$\Delta L = L_{\text{gen}}(\theta^*) - L_{\text{gen}}(\theta_0) \gg 0$$ where $L_{\text{gen}}(\theta)$ is the expected generalization loss over $D_{\text{gen}}$ (Ding et al., 11 Jun 2025, Lai et al., 7 Jul 2025). The challenge is most severe when neither old data nor explicit task boundaries are accessible, demanding solutions that are computationally practical and memory efficient.

2. Core Algorithms and Mechanistic Variants

A spectrum of CSFT frameworks has been introduced to address forgetting, each tailored to domain and constraints:

2.1 Non-Parametric Feature Transformation (FeTT)

FeTT (Qiang et al., 20 May 2024) implements CSFT in class-incremental learning for vision by (a) applying one-time parameter-efficient fine-tuning (PEFT) to a pre-trained backbone on initial task data, then (b) freezing all backbone parameters, and (c) introducing a non-parametric, channel-wise feature transformation $T(\cdot)$ to the concatenated outputs of adapted and original backbones. Two instantiations are

$$T_{\log}(x) = \frac{1}{\ln^\eta(1/x + 1)}, \qquad T_{\text{pow}}(x) = x^\kappa,\ \eta, \kappa > 0$$

This approach compensates for class-marginal distribution deviation in later tasks and alleviates channel suppression induced by initial fine-tuning. The classification head is replaced by a nearest-prototype scheme in the transformed feature space; no gradient updates occur after the first task. Empirical results on CIFAR100 and VTAB show final accuracy increases of 2–3 points over PEFT baselines and up to 1–2% further via backbone ensemble (FeTT-E), which independently applies FeTT to diverse PTMs.

2.2 Instruction Distribution Reconstruction and Synthetic Mixing

To address SFT in LLMs absent original data, synthetic input–output pairs are constructed by first sampling instruction candidates from the pre-trained model, then ensembling multi-model responses (e.g., Llama-3, GPT-4, Qwen), and filtering candidates for response plausibility (Ding et al., 11 Jun 2025). The synthetic dataset $D_{\text{syn}}$ is mixed with new task data at a controlled ratio ($\alpha \approx 0.83$), and fine-tuning proceeds on the union: $$D_{\text{mix}} = D_{\text{syn}} \cup D_{\text{dom}}$$ This preserves general capability (average accuracy 39.21% vs. 39.0% for base model) and matches task-specific gains, outperforming alternatives using naive replay or public datasets.

2.3 Core Parameter Isolation and Spherical Fusion (CPI-FT)

CPI-FT (Wang et al., 29 Aug 2025) identifies per-task "core parameters" by probe fine-tuning for a few epochs on each task and measuring per-weight update magnitude. Core sets $C_i$ for each task are selected as the top $p\%$ by absolute update: $$C_i = \text{arg\,topk}_{j}( |\theta_j^{(i)} - \theta_j^{(0)}| ),\ p=1\%$$ Tasks are grouped by Jaccard overlap among $C_i$, allowing parameter fusion via direct transplantation for core parameters and Spherical Linear Interpolation (SLERP) for non-core parameters. Final lightweight multi-stage SFT is scheduled, with dynamic freezing of prior core regions to prevent overwriting. CPI-FT reduces forgetting by 65–80% compared to baseline SFT, and improves average normalized score on diverse tasks by $\sim$0.5 points.

2.4 Block Parallelization and Hidden State Interpolation

Control LLM (Wei et al., 19 Jan 2025) remediates catastrophic forgetting in LLM CSFT by parallelizing each transformer layer into a frozen pretrained block and a trainable expanded block. The forward pass linearly interpolates hidden states: $$h_\text{combined} = (1-\alpha) h_\text{pre} + \alpha h_\text{exp}$$ with additional divergence loss (MSE) between branches. This permits the expanded branch to absorb new knowledge while the frozen branch preserves legacy capabilities. On Llama-3.1-8B-Instruct, this yields $+14.4\%$ Math-Hard and $+10\%$ MBPP-plus uplift, with $<4.3\%$ degradation on MMLU (versus $>35\%$ for full-parameter SFT).

2.5 Selective Freezing and Soft-Mask Gradient Routing

CoFiTune (Zhang et al., 16 Apr 2024) performs a two-stage control: (a) empirical tree search identifies a module (e.g. FFN) in a contiguous layer range for tuning, freezing all other modules; (b) a fine-grained soft-mask $M$ is computed for each parameter inside this module based on importance to versatility (by KL-divergence sensitivity with dropout). Gradients are adaptively down-scaled: $\hat\nabla_m = (1-\mathbf{I}_m) \odot \nabla_m$ This yields a $14\%$ versatility improvement at 13B scale, with only marginal speciality loss.

3. Implementation Details and Algorithmic Schematics

The following table summarizes key workflow primitives for major CSFT algorithms:

Approach	Parameter Adaptation	Forgetting Mitigation
FeTT	1-shot PEFT, freeze	Channel transform $T(\cdot)$, no replay
CPI-FT	Core-parameter probe, transplant	Dynamic freezing, SLERP for non-core
Control LLM	Block-level fork, interpolate	Hidden state alignment, partial tuning
CoFiTune	Freeze & tune mid-range modules	Soft-masked gradients by versatility importance
Instruction Mix (Ding et al., 11 Jun 2025)	Standard SGD	Synthetic replay, multi-model vote, response scoring

Each algorithm includes detailed pseudocode in source papers; Control LLM and CPI-FT require custom backward-hooks for gradient or parameter management, while FeTT and CoFiTune can be implemented as wrappers around standard forward/backward passes.

4. Empirical Performance and Comparative Metrics

Quantitative evaluation of CSFT variants is based on average (or continual) accuracy across tasks, a forgetting measure (typically final vs. maximal past accuracy), and specialized task metrics (e.g. exact match for coding/math, GPT-4 score for instructions):

• FeTT on CIFAR100 (Inc5): raises avg-acc 87.57% $\to$ 89.22%, final-acc 81.26% $\to$ 83.42%.
• Instruction Mixing CSFT recovers base model performance on MMLU/Math; average 39.21% vs. 39.0% for base, outperforming replay-based SFT.
• CPI-FT (LLaMA-2-7B): AvgNorm 7.21 vs. 6.75 (multi-stage), $-5.7$ forgetting (A$\to$B), $+12.2$ on new task.
• Control LLM: Math-Hard $+14.4\%$ uplift, coding pass@1 $+10\%$, with forgetting bounded to $<4.3\%$ degradation on held-out tasks.
• CoFiTune: On Finance (13B), Versatility increases from 0.403 (Full SFT) $\to$ 0.506, while Spec. is marginally higher.

CSFT baselines without structural constraint typically exhibit steep accuracy declines (up to $>35\%$) on original domains after 20–40K updates (Wei et al., 19 Jan 2025, Lai et al., 7 Jul 2025). Ensemble-based or synthetic data replay methods generally provide additional points of robustness and accuracy.

5. Limitations and Recommendations

Despite progress, standard CSFT strategies remain susceptible to interference when task relatedness is low or when domain shifts are abrupt (Lai et al., 7 Jul 2025). Pure SFT (cross-entropy only) forgets heavily; KL-based regularization or simple curriculum is insufficient. Theoretical analysis highlights the necessity for implicit or explicit regularization that scales effective gradients per instance or parameter by a task-confidence or surrogate reward variance. Models with adaptive learning rates, data-driven filtering, and parameter isolation provide superior retention without sacrificing adaptation.

To maximize stability:

• Prefer parameter-efficient or structurally isolated fine-tuning for CL/CPT scenarios with incremental tasks.
• Apply replay via synthetic data or ensemble agreement scoring when original data is inaccessible.
• Use per-parameter or per-sample importance estimates to restrict destructive drift in critical weights.
• In transformer architectures, prioritize mid-depth FFN tuning and freeze lower layers for generalizability (Zhang et al., 16 Apr 2024).

6. Practical Applications and Deployment Considerations

CSFT is instrumental in continually evolving on-device AI, LLM deployment for domain-specialized chat or retrieval, vision models for novel class acquisition, and enterprise NLP systems requiring privacy or efficiency guarantees. Some approaches (e.g. Control LLM) have seen production use in large-scale industrial settings, demonstrating minimal regression in general benchmarks and state-of-the-art performance in newly introduced domains with competitive compute and data budgets (Wei et al., 19 Jan 2025). Straightforward backwards hooks, selective parameter freezing, or feature transformations can be implemented in standard deep learning frameworks for scalable deployment.

7. Current Controversies and Open Research Questions

There is increasing evidence from comparative works (Lai et al., 7 Jul 2025) that reinforcement fine-tuning and other non-SFT paradigms may offer superior knowledge retention in continual post-training, owing to their implicit, variance-aware updates. Purely supervised methods as commonly practiced are rarely sufficient for long-term retention unless combined with isolation, filtering, or replay. How to optimally partition parameters for isolation, quantify per-task "distance," and design architectures robust to abrupt non-i.i.d. task streams remain open research directions. Direct empirical comparison across modalities, and integration of synthetic and structural techniques, are active topics of inquiry.