Multi-Task Instruction Tuning

• Multi-task instruction tuning is a supervised paradigm that reformulates heterogeneous tasks into unified instruction-response pairs, enabling shared parameter learning across domains.
• It leverages diverse instruction designs, data augmentation, and dynamic task weighting to balance optimization challenges and mitigate gradient interference.
• Empirical results show enhanced sample efficiency and cross-task transfer in NLP, multimodal systems, and specialized areas, with performance gains tied to instruction quality and task compatibility.

Multi-task instruction tuning is a supervised adaptation paradigm in which a single model is fine-tuned on a mixture of tasks recast as natural-language instruction–response examples, typically in a unified text-to-text or seq2seq format. Across the literature, the paradigm appears in pure NLP, multimodal vision-language systems, graph-LLMs, molecular modeling, and clinical AI, with the shared objective of learning instruction following, cross-task transfer, and schema unification under parameter sharing rather than training isolated task-specific systems (Xu et al., 2022). In its simplest form, each training instance pairs an instruction and task input with a target output, and optimization minimizes conditional generation loss, e.g. $$\mathcal{L} = -\sum_{(\mathbf{x}, \mathbf{y}) \in \mathcal{D}} \log P(\mathbf{y}\mid \mathbf{x};\theta)$$, where $\mathcal{D}$ is the union of instructionized task datasets (Wang et al., 2022).

1. Formalization and core objective

A defining property of multi-task instruction tuning is the conversion of heterogeneous tasks into a common conditional-generation interface. In UnifiedABSA, 11 aspect-based sentiment analysis tasks are mapped to a Unified Sentiment Instruction containing task name, sentiment options, category options, and an output template; all tasks are then optimized jointly through a single text-to-text backbone (Wang et al., 2022). In InstructUIE, named entity recognition, relation extraction, and event extraction are reformulated as instruction, options, and text $\rightarrow$ natural-language output, enabling a single 11B FlanT5 model to cover 32 datasets in a unified framework (Wang et al., 2023).

This unification is not merely syntactic. It is intended to expose inter-task dependency under shared parameters. UnifiedABSA explicitly attributes its gains to unified modeling and multi-task learning over related subtasks such as aspects, opinions, categories, and sentiments (Wang et al., 2022). InstructUIE likewise uses a common output schema—such as “EntityType: EntitySpan” for NER or “RelationType: HeadEntity, TailEntity” for RE—to induce transfer across extraction problems that are traditionally trained separately (Wang et al., 2023).

The same principle holds in scenario-focused settings. For writing assistance, seven tasks—grammaticality, fluency, clarity, coherence, simplification, neutralization, and paraphrasing—are reframed in a single Alpaca-style template:

$\mathcal{D}$1

The study shows that task-specific instruction tuning can improve a focused application even when the aim is not broad general-purpose instruction following (Zhang et al., 2023).

2. Instruction design and dataset construction

The effectiveness of the paradigm depends heavily on how instructions are authored, diversified, and aligned with task structure. MultiInstruct established an early multimodal benchmark with 62 tasks from 21 datasets, organized into 10 broad categories, each equipped with 5 expert-written instructions in a unified seq-to-seq format (Xu et al., 2022). The benchmark also introduced the Sensitivity metric to quantify performance variance across equivalent instructions, showing that more tasks and more instruction variants reduce sensitivity to wording (Xu et al., 2022).

Subsequent work pushed instruction diversification toward automation. INSTRAUG begins from a small set of basic meta-instructions, uses LLM-based paraphrastic expansion with Placeholder-Protected Generation, and expands multimodal instruction data by up to 30 times; on MULTIINSTRUCT and InstructBLIP, the augmented instructions improved alignment across 12 multimodal tasks (Han et al., 2024). The mechanism is notable because it targets instruction diversity rather than raw instance count, and the reported result that a 59K instance set with augmented instructions can outperform a 564K set with non-augmented instructions suggests that phrasing coverage can function as a scaling variable in its own right (Han et al., 2024).

In multimodal bilingual training, Ziya-Visual constructs instruction data by translating gathered English image-text datasets into Chinese with GPT-4 and generating instruction–response pairs through in-context learning. Its data format is explicitly structured as [System Message] [User Instruction/Input (text/bboxes)] [Assistant Response], and augmentation uses rewriting strategies labeled Deepening, Concretizing, Increasing Reasoning, and Adding Constraints (Lu et al., 2023). In dialogue, context-dependent instruction tuning goes further by generating per-example instructions conditioned on dialogue history, because fixed task-level instructions were found ineffective for multi-turn response generation (Kwak et al., 2023).

Synthetic data generation also appears outside language and vision. PEIT first aligns textual descriptions, SMILES, and biochemical properties to synthesize instruction triplets, then converts them into instruction–response pairs for molecule captioning, text-based molecule generation, property prediction, and multi-constraint molecule generation (Lin et al., 2024). This suggests that multi-task instruction tuning is increasingly coupled to upstream data-synthesis systems rather than relying solely on hand-annotated instruction corpora.

3. Optimization, task balancing, and interference mitigation

Once tasks are mixed, optimization becomes a central issue because naïve averaging can amplify data imbalance, token-length bias, and gradient conflict. CoTBal formulates multi-task visual instruction tuning through a Generic Task Weighting loss,

$$L_{\text{GTW}} = \sum_{i=1}^N \alpha_i \frac{1}{S_i}\sum_{j=1}^{S_i}\frac{1}{T_{ij}}\sum_{k=1}^{T_{ij}} -\log p(t_{ijk}),$$

and defines two balancing axes: Inter-Task Contribution and Intra-Task Difficulty. Tasks receive higher weight when they contribute strongly to others, receive little help from others, or are intrinsically difficult (Dai et al., 2024). The paper reports superior and more balanced overall performance on three visual-instruction benchmarks (Dai et al., 2024).

A different line of work frames the problem as cross-task interference in shared parameters. BADIT argues that interference persists even in task-isolation and mixture-of-experts approaches because many parameters remain shared. It decomposes pre-trained weight matrices into orthogonal high-singular-value LoRA experts,

$$\mathbf{W}^0 \approx \sum_{k=1}^{K}\alpha_k \mathbf{A}_k\mathbf{B}_k + \widehat{\mathbf{W}},$$

and dynamically restores expert orthogonality through spherical clustering of rank-1 components (Wang et al., 7 May 2026). On SuperNI with six LLMs, BADIT outperforms the strongest prior method by an average of +2.68 ROUGE and maintains near-$90^\circ$ separation between expert gradients during training (Wang et al., 7 May 2026).

Budgeted settings motivate yet another formulation. ADAPT learns task sampling proportions under explicit token budgets by maintaining task logits $\mathbf{w}$, defining $p_i=\exp(w_i)/\sum_j\exp(w_j)$, and updating them via meta-gradients of a smooth worst-case validation objective with entropy regularization (Kadasi et al., 4 Dec 2025). On 20 Natural Instructions task types and three $\sim$1B open-weight LLMs, ADAPT matches or slightly improves the best static mixture while reaching the best supervised validation loss using only 4–38% of the tokens required by supervised fine-tuning baselines (Kadasi et al., 4 Dec 2025).

These methods share a common diagnosis: multi-task instruction tuning is not only a data-formatting exercise, but also a multi-objective optimization problem in which task scheduling, weighting, and subspace allocation materially affect transfer and stability.

4. Architectural realizations across modalities

Although the earliest and simplest instantiations use text-to-text models such as T5, BART, FlanT5, or LLaMA derivatives, the paradigm now spans several architectural families. In multimodal instruction tuning, MultiInstruct fine-tunes OFA on instruction-plus-image inputs in a unified seq-to-seq format, with image or region information converted into tokens (Xu et al., 2022). CM3Leon extends the paradigm to a decoder-only token-based multimodal LLM in which instruction sequences may interleave text and image tokens separated by <break> markers, and the same next-token objective is retained through supervised fine-tuning (Yu et al., 2023).

Vision-language chat models often add an explicit alignment module. Ziya-Visual uses a Vision Transformer, the BLIP-2 Querying Transformer (Q-Former), a bilingual Ziya-LLaMA-13B backbone, and in its chat variant applies LoRA to both ViT and LLM modules. The Q-Former serves as a bottleneck that projects visual features into the LLM embedding space, and in instruction-aware settings it can also receive the instruction text (Lu et al., 2023). Continual multimodal instruction tuning studies build on related architectures such as BLIP2 and InstructBLIP, but reinterpret the projection layer and adapters as the locus of continual adaptation (He et al., 2023).

Other domains require different encoders while preserving the same instruction-tuning logic.

Domain	Representative system	Alignment mechanism
Vision-language	Ziya-Visual	Q-Former + projection into LLM space
Graph-language	UniGraphLM	Multi-domain, multi-task GNN encoder + learnable projector
Molecules	PEIT-LLM	Multimodal pretraining over text, SMILES, properties + synthesized instructions
Clinical radiology	OmniFM-DR	Image encoder + text encoder + unified seq2seq decoder

UniGraphLM illustrates how far the abstraction can be generalized. It combines a multi-domain, multi-task GNN encoder with alignment instruction tuning so that node-, edge-, and graph-level representations are mapped into the LLM token space and consumed as graph tokens in an instruction sequence (Chen et al., 12 May 2026). OmniFM-DR likewise converts classification, localization, segmentation, and report generation for chest radiographs into homogeneous sequence outputs from an image-plus-instruction input, using a unified transformer and a dataset of 13.4 million instruction and ground-truth pairs (Xu et al., 2023).

A plausible implication is that the central architectural requirement is no longer a specific backbone class, but the existence of a stable interface through which heterogeneous task evidence can be serialized or projected into a common autoregressive or seq2seq supervision stream.

5. Empirical properties: transfer, sample efficiency, and specialization

A recurring empirical claim is that multi-task instruction tuning improves transfer and sample efficiency, though the gains are task-dependent. On 119 unseen SuperNI tasks, instruction-tuned Tk-Instruct 3B in the multi-task setting matches the previous supervised SOTA with only 6% of downstream training data, and with 100% of the data reaches ROUGE-L 74.68, a 3.69-point improvement over the previous SOTA (Gupta et al., 2023). In the single-task setting, 25% of downstream data is sufficient to surpass SOTA (Gupta et al., 2023).

Low-resource improvements appear in specialized domains as well. UnifiedABSA reports average F1 improvements of 2.17% on Restaurant-ACOS and 1.24% on Laptop-ACOS in the full-supervised setting, and about 3–6% F1 gains in 32-shot and 64-shot settings across 11 tasks, while requiring only one model instead of 11 dedicated models (Wang et al., 2022). In clinical information extraction, leave-one-dataset-out multi-task instruction tuning markedly improves zero-shot and few-shot transfer: for GatorTronLlama, zero-shot F1 rises from 0.0155 to 0.3596 for clinical concept extraction and from 0.0189 to 0.3154 for clinical relation extraction, with further rapid gains from 5–20 labeled examples (Peng et al., 5 Sep 2025).

The paradigm also helps specialist models under some conditions rather than only producing generalists. “Specialist or Generalist?” finds that generalist-then-specialist tuning consistently improves performance when task coverage is broad, with the largest gains in low-resource settings; for example, in a classification setting with 2k specialist examples, adding GPT4-Instruct raises accuracy from 65.36% to 71.31%, whereas gains are minimal at 10k examples and may disappear for narrow tasks such as Yelp sentiment classification (Shi et al., 2023). The same study reports that generalist data improves understanding and reasoning, but factual knowledge can degrade when the generalist corpus contains hallucinated content (Shi et al., 2023).

In multimodal systems, the benefits are similarly concrete. MultiInstruct shows that transfer from the text-only Natural Instructions dataset can improve overall zero-shot multimodal performance when combined properly, though text-only tuning alone can cause the model to ignore image tokens (Xu et al., 2022). Ziya-Visual reports competitive performance on English image-text retrieval, image captioning, and visual question answering, while GPT-4-based leaderboard evaluation indicates satisfactory Chinese multimodal dialogue capability (Lu et al., 2023). CM3Leon shows that a second multi-task supervised fine-tuning stage yields strong controllability for image editing, image-controlled generation, segmentation, and image-to-text tasks, while maintaining high text-to-image performance (Yu et al., 2023).

6. Limitations, misconceptions, and active research directions

A common misconception is that multi-task instruction tuning reliably helps every task. Several studies qualify this. On SuperNI categories, instruction tuning hurts question rewriting and title generation relative to baselines (Gupta et al., 2023). In scenario-specific NLP, adding broad generalist data offers limited or negative benefit for narrow-coverage tasks that are already near saturation, and hallucinated generalist responses can damage factual QA performance (Shi et al., 2023). For writing assistance, fine-tuned LLaMA models improve average task performance, but single-task deployment can still favor leaner specialist systems because LLaMA-13B-GEC requires about 397 GPU hours of training and 0.7 samples/sec inference, compared with about 21 GPU hours and 274.7 samples/sec for RoBERTa-Large (Zhang et al., 2023).

Another misconception is that instruction unification eliminates forgetting and interference. Continual instruction tuning for large multimodal models shows that catastrophic forgetting persists in both BLIP2-initialized and jointly instruction-tuned initializations, although forgetting is milder when the starting point already has multi-task instruction tuning (He et al., 2023). Replay methods and model expansion strategies such as EProj are consistently effective, and the paper’s task-similarity-informed regularization and model expansion methods further improve continual performance (He et al., 2023). In multimodal continual instruction tuning, BranchLoRA attributes forgetting partly to parameter inefficiency in MoELoRA and proposes an asymmetric design with shared $\mathbf{A}$, expert-specific $\mathcal{D}$0, flexible tuning-freezing, task-specific routers, and an inference-time task selector; on the CoIN benchmark with LLaVA-1.5-7B, it improves ACC from 37.13 to 44.20 and MAA from 42.76 to 49.94 while reducing trainable parameters from 350M to 222M (Zhang et al., 31 May 2025).

Current directions therefore emphasize controllable mixture design, continual adaptation, and better instruction quality rather than assuming that larger task unions are automatically beneficial. This suggests a more precise view of multi-task instruction tuning: it is a family of alignment methods for shared models, whose success depends on instruction diversity, task compatibility, optimization geometry, and the costs of maintaining broad competence without sacrificing specialized fidelity.