Sequential Multi-task Fine-tuning

• Multi-task sequential fine-tuning is a strategy that adapts pre-trained models to ordered tasks, reducing negative transfer and mitigating catastrophic forgetting.
• It leverages stage-wise techniques like meta fine-tuning, task-type filtering, and expert ensembles to enhance specialization while maintaining strong performance.
• This approach is ideal for incremental task scenarios, ensuring efficient deployment across domains such as NLP, LLMs, computer vision, and robotics.

Multi-task sequential fine-tuning is an approach for optimizing neural models—especially large-scale pre-trained models—across a set of related downstream tasks by introducing an ordering or explicit multi-phase pipeline in the adaptation of model parameters. This paradigm contrasts with simultaneous multi-task fine-tuning, where all tasks are seen together, and is designed to exploit both inter-task transfer and selective specialization, reduce negative transfer, and mitigate catastrophic forgetting. Sequential fine-tuning is particularly relevant for environments where tasks arrive or must be adapted incrementally, or where data and computational constraints preclude joint training. Contemporary instantiations include meta fine-tuning in NLP, task-type-filtered staged adaptation in LLMs, ensemble-of-experts pipelines in continual learning, as well as domain- and modality-specific frameworks for computer vision, medical imaging, and robotics.

1. Core Principles and Motivation

Sequential multi-task fine-tuning proceeds by initializing a model with pre-trained weights and then adapting it to a collection of tasks in a staged or ordered manner. At each stage, the model is either adapted jointly across a curated subset of tasks, or continually refined, leveraging information from earlier tasks to inform learning on new ones. The main motivators are:

• Negative Transfer Mitigation: Simultaneous multi-task learning often suffers from destructive interference, especially when tasks are heterogeneous (e.g., classification vs. generation). Sequentialization enables task grouping, type filtering, or meta-learning steps that reduce such conflicts (Qu et al., 2024).
• Catastrophic Forgetting Prevention: Approaches such as expert ensembles with rehearsal, replay buffers, or knowledge distillation anchor prior knowledge, retaining earlier-task competence as new tasks are incorporated (Wang et al., 9 Apr 2025, Cai et al., 2023, Ye et al., 7 Sep 2025).
• Knowledge Specialization and Positive Transfer: Ordered adaptation, often with meta-learning or domain-invariant representation training, aims to maximize generality while facilitating efficient specialization and improving few-shot/few-task adaptation (Wang et al., 2020, Bigoulaeva et al., 2022).
• Deployment and Resource Constraints: Sequential pipelines (e.g., three-stage LLM adaptation) can achieve near single-task accuracy across many tasks while vastly reducing model-per-task deployment cost, crucial for online server deployment or edge inference (Qu et al., 2024).

2. Representative Methodologies

2.1 Meta Fine-Tuning (MFT – NLP, Text Mining)

The MFT paradigm (Wang et al., 2020) consists of two stages: (i) meta fine-tuning over a pool of related domains, extracting highly transferable, domain-invariant features via an instance-weighted multi-task loss and a skip-layer domain-corruption auxiliary objective; (ii) subsequent per-domain fine-tuning initialized from the meta-learned parameters. This pipeline consistently outperforms both single-domain adaptation and classic adversarial/domain-adversarial multi-task training, particularly in few-shot regimes.

Key elements:

• Typicality Weighting: Each instance contributes to the meta-loss proportionally to its representativeness, blending intra- and inter-domain prototype similarity.
• Domain Corruption Loss: Forces shared representations at select network layers to obfuscate domain cues, yielding robust generalization.
• Sequential Transition: Post-meta fine-tune, domain-specific adaptation is run as independent fine-tuning steps with better initialization.

2.2 Staged Sequential Fine-tuning for LLMs

A three-stage pipeline is employed for LLM adaptation in real-world online serving (Qu et al., 2024):

1. Task-Type Filtering: Tasks are filtered by output type (e.g., only fixed-label classification tasks retained) to preclude negative interference.
2. High-Resource Task Specialization: Model is fine-tuned solely on high-resource tasks, ensuring these reach convergence before introducing potential low-resource overfitting.
3. Full Mixture Fine-Tuning: All tasks are reintroduced, but mixture coefficients are capped and temperature-scaled to prevent rare tasks from dominating and to permit early stopping per-task.

This approach allows unified LLMs to reach ≥99% of the per-task baseline accuracy on nearly all tasks while reducing model deployment overhead by up to 90.9%.

2.3 Sequential Ensemble of Experts (Continual LLM Fine-tuning)

The SEE framework (Wang et al., 9 Apr 2025) incrementally adds expert modules as new tasks arrive. Each expert is a LoRA-adapted instance of the base model, trained with both current-task examples and negative examples (previous-task queries labeled as abstentions). Routing is handled sequentially—without a central router—by specialized tokens. This distributed scheme, combined with rehearsal and task-specific adapters, achieves near-forgetting-free lifelong learning and superior OOD generalization compared to standard rehearsal or vanilla multi-task learning.

2.4 Task-Attentive Transformers and Replay with Knowledge Distillation

For bimodal (vision and language) continual learning, task-attentive architectures (Cai et al., 2023) dynamically allocate task tokens and classifier heads, freezing prior parameters, while anchoring the shared network using intermediate knowledge distillation and minimal replay. This achieves low forgetting rates (e.g., 11.7% vs 31.0% for classical replay) and maintains or surpasses absolute task accuracy when stacking domain-diverse tasks.

2.5 Sequential Instruction Tuning

For compositional tasks (e.g., cross-lingual QA, vision-language QA), sequentially augmenting instruction-tuning datasets with multi-step (chained) prompts enables LLMs to follow interdependent instructions within a single query—substantially improving performance on sequence-following benchmarks and demonstrating transfer even when only trained on dummy or canonical templates (Hu et al., 2024).

3. Empirical Results and Comparative Insights

Published results reveal:

Paradigm	Domain	Example Tasks	Forgetting/Retention	Typical Gain over Baselines	Deployment Impact
MFT	NLP	MNLI, Amazon, Taxonomy	Forgetting not major	+2.4–3.0% on accuracy	N/A
3-stage LLM	LLMs	CLUE, Multitask App	N/A	≥99% of per-task baseline	80–91% reduction in overhead
SEE	LLMs	SuperNI (seq. CL), MMLU	BWT≈0	Matches/exceeds multi-task/rehearsal	Low latency increase; OOD robust
TAM-CL	Vision-Lang	SNLI-VE, COCOQA, PathVQA	Forgetting 11.7%	State-of-the-art on all tasks	O(H)/task; moderate runtime
MedSeqFT	Medical	3D CT/MRI segmentation	Forgetting mitigated	+3.0% Dice (CT), +1.4% Dice (MRI)	+5.9h cost (5 tasks, 27h total)

If trained carelessly (e.g., single-step instruction tuning for multi-step tasks; naive simultaneous multi-tasking of heterogeneous objectives), models perform substantially worse or display severe overfitting or negative transfer (Hu et al., 2024, Bigoulaeva et al., 2022).

4. Technical Implementations and Loss Structures

Across methodologies, sequential pipelines are instantiated with precisely defined loss compositions and protocol steps:

• Meta Fine-Tuning (Wang et al., 2020):
  • Stage I: Weighted classification and domain-corruption loss, iterated for 1–2 epochs.
  • Stage II: Per-domain cross-entropy fine-tuning.
• SEE (Wang et al., 9 Apr 2025):
  • Each expert trained via cross-entropy over both indicator tokens (∈{pos,neg}) and output.
  • Negative samples for rehearsal sampled at 1–20% of historical queries.
• MedSeqFT (Ye et al., 7 Sep 2025):
  • Sequential full fine-tune plus LoRA-based knowledge distillation using maximum data similarity-selected buffers to preserve general and pre-trained representations.
• Instruction Chaining (Hu et al., 2024):
  • Loss per subtask in chain, sum over cross-entropies for all subtasks.

Temperature scaling, mixture capping, and buffer-based knowledge retention are used widely to tune the degree of exposure per task and preserve information from earlier stages.

5. Practical Considerations and Limitations

• Hyperparameter Sensitivity: Buffer sizes, KD weights, capping ratios, and staged schedules critically affect knowledge retention and transfer performance (Ye et al., 7 Sep 2025, Qu et al., 2024).
• Overfitting Risks: Sequential pipelines may over-specialize to new tasks unless bolstered by replay, KD, or meta-learning (Bigoulaeva et al., 2022, Cai et al., 2023).
• Compute and Memory Footprint: The addition of adapters, token/head expansions, or buffer sampling introduces minimal to moderate overhead (e.g., O(H) or O(tasks × d), where H is layer width, d is embedding size).
• Data Accessibility: Joint multi-task methods require full data availability, while sequential methods excel when tasks emerge incrementally or data cannot be pooled.
• Latency: In expert-ensemble models such as SEE, inference time scales sublinearly with the number of experts, and can be a minor factor compared to sequential input/output length in instruction-chaining settings.

6. Domain-Specific Extensions and Future Directions

Recent work demonstrates that:

• Autonomous Robotics (ExT framework): Multi-task pretraining enables few-shot acquisition of new skills and robust, KL-regularized RL adaptation to OOD conditions, while replay/interleaved SFT prevents catastrophic skill loss (Zhai et al., 18 Sep 2025).
• Medical Imaging: Sequential FT with knowledge distillation and maximum data similarity indexing (MedSeqFT) enables both high slice-wise segmentation retention and superior transfer to novel structures or rare pathologies (Ye et al., 7 Sep 2025).
• Vision-and-Language CL: Dynamic parameter expansion with task-attentive tokens, layer freezing, and intermediate knowledge distillation achieves superior scaling and accuracy (Cai et al., 2023).
• Prompt-Driven and Modality-Mixed Extensions: Ongoing research explores prompt-augmented, buffer-managed, multi-modal, and highly adaptive pipelines for complex real-world deployment settings (Ye et al., 7 Sep 2025, Zhai et al., 18 Sep 2025).

7. Comparative Analysis and Future Research

Consistent empirical evidence supports the superiority of well-calibrated sequential fine-tuning pipelines over both parallel single-task adaptation and naive multi-task baselines when tackling non-i.i.d. task sequences, heterogeneous objective spaces, or when retention and deployment cost are critical. Yet, optimal scheduling, automated buffer management, and extension to truly open-ended continual learning settings remain open research areas.

Effective cross-task transfer in settings where tasks are explicitly dependent (e.g., inference plus explanation) further validates that sequential fine-tuning, with judicious epoch and buffer balancing, can drive state-of-the-art performance—outperforming both hierarchical multi-task learning and single-stage approaches (Bigoulaeva et al., 2022). However, over-specialization and suboptimal schedule tuning can still degrade generalization and retention, emphasizing the need for domain-aware regularization and knowledge anchoring mechanisms.