Small Model Continual Fine-tuning

• Small Model Continual Fine-tuning is the sequential adaptation of compact pretrained models to incorporate new domain knowledge while maintaining existing capabilities in resource-constrained settings.
• It encompasses diverse regimes such as supervised fine-tuning, continued pre-training, and modular adaptation that use strategies like parameter isolation, replay, and regularization to mitigate catastrophic forgetting.
• Practical applications in NL-to-DSL systems and medical NLP highlight trade-offs between domain specialization and general reasoning, demonstrating the balance between performance and resource efficiency.

Small model continual fine-tuning denotes the sequential adaptation of relatively compact pretrained models under tight constraints on latency, memory, compute, privacy, or deployment complexity. In current practice, the term spans several related regimes: supervised downstream updating, continued pre-training on unlabeled domain corpora, replay-assisted continual supervised fine-tuning, and modular or task-isolated adaptation that avoids overwriting shared weights. Across these regimes, the central technical problem is the same: incorporate new domain knowledge or task behavior while preserving previously learned capability and keeping the resource profile characteristic of small backbones or small LLMs (Coleman et al., 18 Apr 2025, Nair et al., 10 Apr 2026, Faroz, 13 Apr 2025).

1. Conceptual scope and learning regimes

The literature distinguishes continual fine-tuning from continued pre-training. Continued pre-training resumes optimization of an already pretrained model on fresh unlabeled domain text in order to update internal knowledge and style, whereas task fine-tuning is supervised optimization for a specific downstream task with labeled data (Faroz, 13 Apr 2025). This distinction matters because the failure modes differ: continued pre-training can improve domain familiarity while degrading general reasoning or instruction adherence, whereas supervised fine-tuning can produce strong task alignment but may forget earlier tasks or overfit when labels are scarce (Ferrazzi et al., 19 Feb 2026, Faroz, 13 Apr 2025).

Evidence across domains shows that continual pre-training is not uniformly necessary. In few-shot intent classification with small PLMs such as BERT-base and RoBERTa-base, direct fine-tuning plus context augmentation and sequential self-distillation outperformed many continual pre-training baselines once at least two labeled examples per class were available (Zhang et al., 2023). In Italian medical NLP, multi-task supervised fine-tuning was the most effective adaptation strategy across 20 clinical NLP tasks, while continual pre-training was only occasionally beneficial and in several cases degraded performance relative to fine-tuning alone (Ferrazzi et al., 19 Feb 2026). By contrast, domain-adaptive continued pre-training of a 125M model on FineWeb-Edu improved MMLU from 0.2304 to 0.2490 and HellaSwag from 0.3816 to 0.4105, but this came with declines on Winogrande and BoolQ, making the specialization trade-off explicit (Faroz, 13 Apr 2025).

A separate line of work argues that continual learning capacity can emerge directly from self-supervised pre-training. Instruction-tuned LLMs initialized from T5-family checkpoints maintained performance on prior tasks under very small rehearsal buffers, whereas randomly initialized models did not, suggesting that broad pretrained representations are a primary source of continual-learning robustness (Scialom et al., 2022). Taken together, these results indicate that “small model continual fine-tuning” is not a single algorithmic recipe. It is better understood as a design space in which the update regime is chosen according to whether the main bottleneck is missing domain knowledge, limited supervision, sequential interference, or production latency.

2. Stability–plasticity mechanisms

The dominant technical theme is the attempt to separate plastic parameters, which absorb new information, from stable parameters, which preserve prior capability. Parameter-efficient fine-tuning is the most common embodiment of this principle. In the PEFT literature, a canonical low-rank update is written as

$W' = W + \alpha A B,$

with only the low-rank factors trained; the broader PECFT literature places LoRA, adapters, prompts, masks, replay, and regularization inside a single continual-learning taxonomy for resource-constrained models (Coleman et al., 18 Apr 2025).

Several mechanisms recur across the literature:

Family	Core mechanism	Representative instances
PEFT and parameter isolation	Update only low-rank, prompt, or masked subsets	LoRA, adapters, prompt tuning, BitFit (Coleman et al., 18 Apr 2025)
Consistency and regularization	Penalize drift on old features, logits, or important weights	EWC, GEM, feature/adapter consistency (Liu et al., 2020, Yu et al., 13 Aug 2025)
Modular or frozen-backbone adaptation	Keep base representation fixed and learn auxiliary modules	input-space tuning, frozen-module composition, task prompts (Marullo et al., 2023, Anderson et al., 2016)

In federated continual learning, “Small Model Continual Fine-tuning” was formalized as a client-side procedure that trains a small CNN and a generator that emits sample-specific LoRA parameters for a frozen ViT. Its continual loss combines feature consistency and adapter consistency on buffered samples,

$$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$

and is paired with cross-entropy on the same buffer, thereby preserving the small model’s ability to regenerate adapter parameters for past data (Yu et al., 13 Aug 2025). In cross-lingual fine-tuning, GEM constrains optimization so that a prior-task loss does not increase while learning a new downstream task, explicitly treating fine-tuning as a two-task continual-learning problem (Liu et al., 2020).

Another stability mechanism targets normalization rather than weights. ConFiT showed that catastrophic forgetting in continual fine-tuning of CNNs is not only a penultimate-representation problem but also an intermediate-layer representational shift problem because batch-normalization statistics become inaccurate. Its Xconv BN stores pre-convolution running means and reconstructs post-convolution means before testing, while hierarchical fine-tuning stages the update sequence as linear probing, BN tuning, and then full fine-tuning to reduce large early changes in convolutional filters (Jie et al., 2022).

An even stronger form of stability is obtained by never modifying the pretrained core at all. Input-space tuning learns only a differentiable transformation $T_\phi(x)$ before a frozen backbone, with the data flow

$$x \rightarrow T_\phi(x) \rightarrow g(\cdot) \rightarrow f_w(\cdot),$$

and achieves continual adaptation by moving the input representation rather than the backbone weights (Marullo et al., 2023). An earlier modular formulation on small-data transfer learning made the same argument more generally: adding trainable modules to frozen pretrained modules preserves prior representational power because the old weights are not overwritten (Anderson et al., 2016).

3. Data curation, replay, and selection

A second major theme is that continual fine-tuning is often limited less by optimizer design than by which new examples are admitted into training. In a production NL-to-DSL system, the core supervised corpus contained 68,000 NL–DSL pairs, including 22,000 expert-curated pairs and 46,000 production-distribution flows converted into NL with GPT-4. A subset of 23,000 examples was further enriched with function signatures, retrieved few-shot examples, step summaries, and a static instruction, and continual updates mixed general data, safety harms, and tenant-specific examples rather than training on the new slice in isolation (Nair et al., 10 Apr 2026). This was a data-mixing strategy for stability even though no explicit anti-forgetting regularizer such as EWC or distillation was used.

On-policy replay pushes this data-centric logic further. Rather than replaying historical gold labels, On-Policy Replay rolls out the current checkpoint on historical prompts, filters generations by task reward or self-confidence, and reuses the surviving prompt–response pairs as ordinary SFT data. Its mixture objective remains standard supervised fine-tuning,

$$L_{\mathrm{OPR}}(\theta) = (1-\lambda)\,\mathbb{E}_{(x,y)\sim D_{\mathrm{cur}}}[-\log p_\theta(y|x)] + \lambda\,\mathbb{E}_{(x,y)\sim B_{\mathrm{replay}}}[-\log p_\theta(y|x)],$$

but the replay distribution is on-policy rather than off-policy (Chen et al., 28 May 2026). On TRACE, for Qwen2.5-7B-Instruct, sequential SFT yielded BWT $-13.93$, while OPR lifted BWT to $-0.65$ at a 10% replay budget and to $-2.29$ at a 1% budget. The same work also showed that low-score replay is uniformly worse than Vanilla Replay, implying that the critical ingredient is the on-policy high-score distribution rather than replay quantity alone (Chen et al., 28 May 2026).

Filtering can also be performed before fine-tuning rather than during replay. A self-adaptive continual instruction-tuning framework used a 0.5B proxy model to compute perplexity-based Instruction-Following Difficulty,

$$\mathrm{IFD}_{ij}=\frac{\mathrm{PPL}(y_{ij}\mid x_{ij})}{\mathrm{PPL}(y_{ij})},$$

along with length and semantic-diversity filters, and only the surviving data were passed to LoRA training of a 14B deployed model (Lin et al., 20 Mar 2025). In the reported medical setting, filtering 30,000 examples down to 10,000 reduced computational cost by 66.7% and slightly improved accuracy from 70.2% to 70.4% relative to training on the full set, while reducing fault cases relative to the untuned model (Lin et al., 20 Mar 2025).

Continued pre-training relies on a related but unlabeled pipeline. The 125M domain-adaptive continued pre-training study emphasized document-length filtering, repetition filtering, hash-based deduplication, unchanged tokenizer vocabulary, and packing to fixed 2,000-token sequences, showing that continual adaptation in small models is also a corpus-engineering problem (Faroz, 13 Apr 2025). In few-shot intent classification, context augmentation adopted an intermediate position: unlabeled GPT-J generations were used for MLM-based adaptation rather than as supervised labels, reducing label drift while still shifting the representation toward the task distribution (Zhang et al., 2023).

4. Production and domain-specific deployments

Small model continual fine-tuning is not merely a benchmark topic; it has become a production strategy when latency constraints make large-model retrieval pipelines unattractive. In the NL-to-DSL case study, a GPT-4 RAG baseline with similarity 0.70 and non-parsed rate 2.1% was replaced by fine-tuned small models such as Mistral-7B, Phi-3-medium, and NeMo-Mistral (Nair et al., 10 Apr 2026). The best Mistral-7B configuration reached similarity 0.72 and non-parsed 1.8% on F2NL-1K, while preproduction authoring latency dropped from P50 13.0s and P90 22.5s for GPT-4 RAG to P50 1.3s and P90 2.35s for the fine-tuned Mistral-7B deployment (Nair et al., 10 Apr 2026). Inference used vLLM with tensor parallelism across eight A100 40GB GPUs, paged attention, continuous batching, KV-cache reuse, and hot-swappable LoRA adapters served through Punica with approximately 2 ms per output token overhead (Nair et al., 10 Apr 2026).

The same system provided an explicit demonstration of continual tenant-specific fine-tuning. Starting from a general model trained on 68K examples, an sAPI-specific test set had similarity 0.06. Incremental fine-tuning on 12 examples gave 0.07; augmentation to 64 examples with signal boosting raised it to 0.53; and adding an explicit sAPI prefix raised it further to 0.65, while general-test similarity stayed in the 0.71–0.73 range and general non-parsed rates stayed around 1.6–1.8% (Nair et al., 10 Apr 2026). The tagged variant increased sAPI non-parsed rate to 8.3%, showing that specialization and parseability can trade off even when average similarity improves. The same continual pipeline also embedded safety behavior: after adding 242 harmful examples to training, harms tests produced similarity 0.0 and non-parsed 100%, with no regression on the general test (Nair et al., 10 Apr 2026).

Medical NLP provides a complementary picture. Across 20 Italian clinical NLP tasks, fine-tuned small models around 1–1.7B parameters were competitive with or superior to much larger prompted baselines. The strongest configuration, Qwen3-1.7B-Instruct plus fine-tuning, achieved a macro average of 63.9, which was 9.2 points higher than Qwen3-32B with 4-shot prompting (Ferrazzi et al., 19 Feb 2026). Few-shot prompting improved zero-shot performance, and the combination of 4-shot prompting and constraint decoding produced a +13.2-point average gain over zero-shot, but 4-shot prompting increased runtime by approximately 53% whereas constraint decoding added negligible overhead (Ferrazzi et al., 19 Feb 2026). This makes the production trade-off explicit: inference-time adaptation is often useful when labels are scarce, but once supervised data exist, fine-tuning is the dominant low-latency strategy.

Continued pre-training remains relevant when labeled data are limited but large unlabeled domain corpora are available. The educational 125M study ran 400M tokens of CPT in about 3 hours on 3 A100 80GB GPUs and extended to 1B tokens in about 4 hours on 6 A100 80GB GPUs, for a total of approximately 33 GPU-hours (Faroz, 13 Apr 2025). The resulting model improved on knowledge-intensive and contextual tasks but lost some general reasoning performance, underscoring that resource-efficient continual adaptation can be economically attractive without eliminating the specialization–generalization trade-off (Faroz, 13 Apr 2025).

5. Modular, routing, and selective alternatives to weight overwriting

A large portion of the literature treats continual fine-tuning not as repeated modification of one parameter set, but as controlled composition of small task-specific components. Sequential Ensemble of Experts exemplifies this view. Each new task trains only a fresh LoRA expert on a frozen base model, and each expert jointly learns routing and response generation through special acceptance and rejection tokens. Queries are offered sequentially from newest to oldest expert, and if all reject, a base model handles the request (Wang et al., 9 Apr 2025). In a 10-task setting on SuperNI with Llama-2-7B, SEE(10%) achieved average ROUGE-L 66.19 and BWT $-0.00$, outperforming multi-task learning at 64.69 and avoiding router retraining (Wang et al., 9 Apr 2025). Its OOD behavior is also explicit: about 99.72% of MMLU instances were routed to the base model with $$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$0 (Wang et al., 9 Apr 2025).

TRACE implements a different decomposition. It first performs a 1-epoch warm-start probe to obtain task-specific update traces, then identifies the top $$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$1 of “core parameters” using L2 norm, Fisher information, or cross-task cosine specificity, and finally updates only the active task’s core subset during continual tuning (Han et al., 29 May 2026). On LLaMA3-8B, the best baseline average across Code, GSM8K, and MedQA was 46.33, while TRACE-CS reached 63.28; on Qwen2.5-14B, the strongest baseline average was 62.34, while TRACE-CS reached 75.39 (Han et al., 29 May 2026). The default $$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$2 is notable because it reframes continual adaptation as sparse parameter discovery rather than continual exposure of all weights to all tasks.

Tangent Model Composition avoids sequential bias by training local tangent deltas around a shared pretrained point and composing them at inference time:

$$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$3

Because composition is linearized around the pretrained anchor, inference costs remain $$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$4 rather than growing with the number of component models. In the reported experiments, TMC improved accuracy by 4.2% relative to ensembling non-linearly fine-tuned models while reducing inference cost by 2.5x to 10x (Liu et al., 2023). The same construction enables zero-cost unlearning by dropping a component delta (Liu et al., 2023).

Prompt-per-task methods achieve similar isolation with even smaller state. In continual table semantic parsing, the C3 framework froze a T5 backbone after an initial adaptation stage and learned a separate prompt for each task, with approximately 0.7M trainable parameters per task, or about 0.05% of T5-base (Chen et al., 2023). Across Spider-Stream and Combined-Stream, Memory Decay was approximately zero because the backbone was frozen and the correct prompt was loaded by schema at inference (Chen et al., 2023). This design is structurally close to earlier modular small-data transfer methods, where frozen pretrained modules and new trainable modules were composed rather than overwritten, ensuring that prior representational capacity remained intact (Anderson et al., 2016).

6. Evaluation, misconceptions, limitations, and unresolved tensions

The topic is evaluated with unusually heterogeneous metrics because the tasks themselves differ. Continual SFT work reports backward transfer,

$$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$5

while vision continual learning often reports average accuracy and forgetting, and domain-specific code generation systems use task-specific structural metrics. In the NL-to-DSL production system, the principal structural score was

$$L_{\mathrm{CL}} = \frac{1}{|B_i^{n-1}|}\sum_{(\hat{x},\hat{y}) \in B_i^{n-1}} \left[\lambda_E \|\hat{E} - \hat{E}'\|_2^2 + \lambda_W \|\hat{W} - \hat{W}'\|_2^2 \right],$$6

alongside the percentage of non-parsed flows, a compiler-oriented validity measure (Nair et al., 10 Apr 2026). Speech continual fine-tuning used WER and also probed retention by measuring loss on the original SSL objective during downstream fine-tuning, arguing that lower forgetting of the pretext task correlates with better generalization (Zaiem et al., 2024). Table semantic parsing used Task Accuracy, Example Accuracy, Initial Accuracy, and Memory Decay (Chen et al., 2023).

Several common misconceptions are contradicted by the current literature. The first is that continual pre-training is always a prerequisite for small-model adaptation. In few-shot intent classification, direct fine-tuning with lightweight augmentation and self-distillation overtook continual pre-training baselines once shot counts rose modestly (Zhang et al., 2023). In Italian medical NLP, fine-tuning was the most reliable strategy, while continual pre-training sometimes hurt instruction-following quality (Ferrazzi et al., 19 Feb 2026). The second misconception is that PEFT by itself solves catastrophic forgetting. Survey work explicitly treats PEFT as efficient but not sufficient: continual settings still require replay, regularization, routing, masking, or isolation to avoid destructive interference (Coleman et al., 18 Apr 2025).

The limitations are similarly recurrent. Small models remain more sensitive to prompt phrasing, unseen compositions, hallucinated operations, and long-context demands than larger models, even after fine-tuning (Nair et al., 10 Apr 2026). Continued pre-training can overspecialize the model: the 125M educational CPT study reported gains on MMLU and HellaSwag but declines on Winogrande and BoolQ in the absence of general-domain replay (Faroz, 13 Apr 2025). Adapter-based continual learning is sensitive to architectural choices: in federated continual learning, modulating deeper ViT blocks led to degradation and variability, whereas shallower blocks were more stable (Yu et al., 13 Aug 2025). On-policy replay depends on score quality; low-score replay was worse than Vanilla Replay, and reward misspecification remained an acknowledged failure mode (Chen et al., 28 May 2026). Expert-based schemes reduce forgetting but accumulate parameters and add sequential routing latency as the number of experts grows (Wang et al., 9 Apr 2025). Xconv BN is specific to BN-equipped convolutional models with task identities available at test time (Jie et al., 2022).

A plausible implication is that the field is converging on hybrid systems rather than one universal algorithm. The evidence supports small-model continual fine-tuning when adaptation is selective, data-aware, and deployment-driven, but it also indicates that stability comes from different sources in different regimes: replay in continual SFT, frozen-backbone modularity in prompt or expert methods, statistical correction in convolutional continual fine-tuning, and careful corpus or sample filtering in data-centric pipelines. The remaining open questions, already visible in the cited work, concern execution-aware evaluation for structured generation, consolidation of growing task modules, robust task-parameter transfer across scales, and methods that preserve general capabilities without sacrificing the latency and simplicity that make small models attractive in the first place (Nair et al., 10 Apr 2026, Han et al., 29 May 2026, Coleman et al., 18 Apr 2025).