Text-Based Compositional Multi-Tasking

• Text-Based Compositional Multi-Tasking is a modeling approach that decomposes complex tasks into reusable text components for improved cross-task generalization.
• It employs diverse formulations like prompt configurations, meta-networks for dynamic parameter generation, and staged fine-tuning to tackle limitations of standard multitask learning.
• Empirical evaluations show notable gains in in-domain and zero-shot performance, highlighting the importance of explicit structural decomposition in complex text tasks.

Text-based compositional multi-tasking denotes a family of modeling and evaluation settings in which text systems are required to solve tasks assembled from reusable components rather than treated as monolithic dataset labels. In the cited literature, composition is expressed at several levels: as prompt prefixes that declare task type and input/output structure, as stagewise decompositions of a complex task into component tasks, as meta-networks that generate task-specific composition parameters, as aspect recombinations in controllable generation, and as sequential execution of multiple text operations within a single inference pass. Across these variants, the central objective is cross-task generalization to unseen combinations of inputs, outputs, controls, or goals (Chen et al., 2022, Bursztyn et al., 2022, Bohdal et al., 21 Jul 2025).

1. Formalizations of compositionality

A recurrent formal distinction in this literature is between task composition, task decomposition, and composition-function sharing. In unified table-to-text modeling, a task instance is specified by a compositional configuration

$$\text{Config} = [\text{Task: } T] [\text{Dataset: } D] [\text{Input: } I_1]\ldots[\text{Input: } I_k] [\text{Output: } O_1]\ldots[\text{Output: } O_m],$$

where $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$, $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$, and $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$. This formulation treats task identity as a composition of declarative primitives rather than a single dataset-name token (Chen et al., 2022).

In compositional fine-tuning, a complex target task $T$ is decomposed into component tasks $\{T_1,\ldots,T_k\}$ with

$T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$

Training is then organized into curriculum phases $C_1,\ldots,C_P$, with phase-$p$ optimization defined by

$$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$

and the reported instantiation sets $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$0 (Bursztyn et al., 2022).

A different formulation appears in Meta Multi-Task Learning, where composition is relocated from prompts to the sequence model’s internal update rule. For task $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$1 and time step $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$2, a shared meta-network generates task-and-position-specific parameters,

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$3

Here the shared object is not a prompt schema but a learned mechanism for producing composition functions dynamically across tasks (Chen et al., 2018).

In on-device compositional multi-tasking, the operative formalization is sequential task application:

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$4

The user supplies a single prompt specifying the desired composition, such as summarization plus translation, and the model must return the final output $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$5 in one forward pass (Bohdal et al., 21 Jul 2025).

These formulations are not interchangeable. A prompt-based configuration, a curriculum over component tasks, a dynamically generated recurrent composition rule, and a sequential multi-operation inference constraint each target a different failure mode of standard multi-task learning. This suggests that “compositional” in this area is best understood as a structural property of how tasks are represented, combined, and reused rather than as a single algorithmic recipe.

2. Compositional task configurations in unified table-to-text models

The most explicit prompt-level realization of text-based compositional multi-tasking is the compositional task configuration framework proposed for unified table-to-text modeling. Building on the text-to-text T5-large encoder-decoder, each training example uses

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$6

with decoder target

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$7

and training maximizes the conditional likelihood

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$8

Examples are mixed via temperature-up-sampling with $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$9, optimization uses AdamW with $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$0 and batch size $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$1, and input length is capped at $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$2 SentencePiece tokens; the configuration prefix uses $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$3 of this budget (Chen et al., 2022).

The declared motivation is that dataset-name prefixes both limit multi-task learning and hinder generalization to new domains or tasks. By instead exposing task type, input types, and output types, the model is forced to factorize its encoder-decoder mapping along those types. The paper’s FeTaQA example uses the encoder prefix [Task: Summarization] [Input: query] [Input: table] [Output: cells] [Output: long answer], with the intended interpretation that the model first produces supporting cells and then synthesizes a long-form answer (Chen et al., 2022).

Empirically, the method is evaluated on ten table-to-text tasks against the UnifiedSKG single-token prefix baseline of Xie et al. With a T5-large backbone, the average in-domain score rises from $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$4 to $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$5, while the zero-shot average rises from $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$6 to $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$7. The reported zero-shot table includes FeTaQA from $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$8 to $$\{I_1 \ldots I_k\} \subseteq \{\text{query}, \text{table}, \text{passage}\}$$9, HybridQA from $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$0 to $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$1, TAT-QA from $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$2 to $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$3, and FEVEROUS accuracy from $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$4 to $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$5. The summarized gains are $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$6 in-domain and $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$7 zero-shot, with the text also highlighting “FeTaQA F1 from $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$8” (Chen et al., 2022).

The ablation studies clarify which configuration fields carry the most compositional signal. On T5-base, removing output types causes the largest zero-shot average drop, from $$\{O_1 \ldots O_m\} \subseteq \{\text{cells}, \text{short-form answer}, \text{long-form answer}, \text{summary}, \text{binary answer}\}$$9 to $T$0; removing task type lowers the average to $T$1 and drives FeTaQA to $T$2; removing dataset name yields $T$3; removing input types yields $T$4. Test-time ablations further show that dropping either [Input: table] or [Input: passage] on hybrid tasks degrades EM or accuracy by $T$5–$T$6 points, and skipping [Output: cells] on FeTaQA reduces zero-shot EM from $T$7 to $T$8 (Chen et al., 2022).

A common misconception is that a unified text-to-text model automatically learns transferable task structure once multiple datasets are mixed. The reported results do not support that view. In this setting, the difference between a monolithic dataset token and a structured declaration of task, input, and output types is associated with a small in-domain change but a much larger zero-shot difference.

3. Dynamic composition functions and Meta Multi-Task Learning

Chen et al. study compositionality at the level of the sequence model’s composition function rather than at the level of prompts. Their starting point is that standard sequence models apply a single fixed composition function at every position and in every task, which is said to induce under-fitting of positional idiosyncrasies and task isolation. The proposed solution is a shared meta-network that learns “how to compose,” while task-specific low-level parameters are generated dynamically for each task and position (Chen et al., 2018).

The implementation uses a Meta-LSTM as the shared meta-network. At each time step,

$T$9

and the meta vector $\{T_1,\ldots,T_k\}$0 constructs the Basic-LSTM parameters through a low-rank factorization. The resulting task-specific Basic-LSTM update is then applied to compute $\{T_1,\ldots,T_k\}$1. For classification, each task has its own output layer with

$\{T_1,\ldots,T_k\}$2

while sequence tagging uses a CRF. The overall multi-task objective is

$\{T_1,\ldots,T_k\}$3

Optimization uses Adagrad with L2-regularization of $\{T_1,\ldots,T_k\}$4, $\{T_1,\ldots,T_k\}$5-dimensional GloVe vectors, batch size $\{T_1,\ldots,T_k\}$6, and, for classification, $\{T_1,\ldots,T_k\}$7, $\{T_1,\ldots,T_k\}$8, $\{T_1,\ldots,T_k\}$9, $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$0, and $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$1 (Chen et al., 2018).

The experimental results separate single-task, multi-task, and transfer settings. On 16 Amazon-domain sentiment datasets, standard LSTM obtains an average $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$2, HyperLSTM $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$3, and single-task Meta-LSTM $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$4. Among multi-task baselines, SSP-MTL reaches $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$5, PSP-MTL $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$6, ASP-MTL $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$7, and Meta-MTL $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$8. The model size is reported as approximately $T = T_k \circ T_{k-1} \circ \ldots \circ T_1.$9M parameters, compared with SSP-MTL at $C_1,\ldots,C_P$0M and ASP-MTL at $C_1,\ldots,C_P$1M. For sequence tagging, POS on WSJ rises from $C_1,\ldots,C_P$2 to $C_1,\ldots,C_P$3, chunking F1 from $C_1,\ldots,C_P$4 to $C_1,\ldots,C_P$5, and CoNLL-2003 NER F1 from $C_1,\ldots,C_P$6 to $C_1,\ldots,C_P$7 (Chen et al., 2018).

The transfer experiment is especially relevant to compositional multi-tasking. In a 16-task hold-one-out setup, the trained Meta-LSTM parameters $C_1,\ldots,C_P$8 are fixed as “off-the-shelf” composition knowledge, and only a new Basic-LSTM is trained for the held-out domain. The average accuracy is $C_1,\ldots,C_P$9, which is $p$0 above the standard single-task LSTM’s $p$1 and competitive with full multi-task models (Chen et al., 2018).

This line of work broadens the meaning of text-based compositional multi-tasking. The compositional object need not be an explicit textual instruction. It can instead be a dynamically instantiated recurrent computation whose meta-knowledge is learned jointly and reused across tasks.

4. Compositional fine-tuning as staged subtask acquisition

Bursztyn et al. formulate composition as explicit decomposition of a complex target task into component text-to-text tasks, followed by stagewise fine-tuning of a pretrained LLM. Their algorithmic summary is “Decompose, Demonstrate, Fine-Tune,” plus inference. Component tasks are manually organized into phases so that tasks in earlier phases are easier and later phases depend on earlier ones. During phase $p$2, all examples from tasks in $p$3 are shuffled and optimized with the standard language-model objective

$p$4

continuing fine-tuning from the previous phase (Bursztyn et al., 2022).

The paper gives concrete decompositions in recommendation and inference domains. In world travel and local dining, the component task types are Factual Statements, Negative Preference Interpretation, Factual Comparisons, and Decision Templates. The examples are intentionally text-based: “The average temperature in Lisbon is” $p$5 “17.5°C”; “You don’t like cold weather.” $p$6 “You like warmer weather.”; “Between London and Lisbon, the city with higher average temperature is” $p$7 “Lisbon”; and the final composite prompt “You don’t like cold weather. Between London and Lisbon, you should visit” $p$8 “Lisbon.” In sports inference, the components are Membership Statements and Act Plausibility, with the final task evaluating the plausibility of a novel sentence in the same format (Bursztyn et al., 2022).

The reported results show that curriculum over component tasks can dominate end-to-end fine-tuning even when the amount of data is held fixed. On the 13B Curie model, world travel decision accuracy is $p$9 for “Decision Only,” $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$0 for “Decision + Facts,” and $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$1 for CFT with all components. Local dining rises from $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$2 and $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$3 to $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$4. Against 8-shot chain-of-thought prompting with DaVinci (175B), world travel is $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$5 for CoT and $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$6 for CFT; local dining is $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$7 for CoT and $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$8 for CFT; sports inference is approximately $$\theta^{(p)} = \arg\min_\theta L_p(\theta), \qquad L_p(\theta)=\sum_{T_i \in C_p}\sum_{(x,y)\in D_i}\ell(f_\theta(x),y)+\lambda\cdot\Omega(\theta,\theta^{(p-1)}),$$9 for CoT and approximately $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$00 for CFT. The summarized claim is that CFT performs at least as well using LLMs only $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$01 of the size (Bursztyn et al., 2022).

The framework also states its own limitations. Decomposition is manual in the original work, automatic decomposition remains a major open problem, the experiments are in English and with GPT-3 engines, and the method is said to be best suited to tasks with a clearly known hierarchical structure. Best-practice recommendations include choosing a small set of meaningful component tasks, ordering them by inference depth, writing multiple natural-language phrasings, balancing data across phases, shuffling within each phase, and optionally adding a regularization term $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$02 to prevent catastrophic forgetting (Bursztyn et al., 2022).

Relative to prompt-based task configurations, compositional fine-tuning shifts the burden of structure from the test-time prompt to the training curriculum. It is therefore a compositional multi-tasking method in which intermediate task competence is deliberately taught rather than inferred solely from parameter sharing.

5. Evaluation regimes, failure modes, and bottlenecks

Several adjacent literatures treat compositional multi-tasking chiefly as a generalization problem. In task-oriented dialogue, the target is a single dialogue that requests two or more tasks in sequence, while training may contain only single-task dialogues or limited multi-task data. Parthasarathi et al. study this setting on MultiWOZ v2.0 with a standard encoder-decoder Transformer trained as a conditional LLM and evaluate BLEU on generated assistant utterances. Synthetic augmentation by random or targeted concatenation of single-task dialogues yields modest gains: the single-task-only model reports multi-task BLEU $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$03 and overall BLEU $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$04, Random_Augment gives $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$05, and Targeted_Augment gives $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$06. Increasing the fraction of real multi-task examples from $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$07 to $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$08, while keeping total dialogues approximately $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$09, raises multi-task BLEU from $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$10 to $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$11. A Domain-Invariant Transformer with a discriminator over single- versus multi-task contexts reaches approximately $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$12 BLEU on multi-task dialogues only when all real multi-task data is used (Parthasarathi et al., 2020).

In multi-aspect controllable text generation, the same issue appears as compositional generalization to attribute combinations never jointly seen in training. CompMCTG evaluates this on four datasets—Fyelp, Amazon Review, YELP, and Mixture—under three protocols: Hold-Out, Attribute Compound Divergence, and Few-Shot. The framework defines an eligible split in which every attribute value present in the compositional test set also appears in the in-distribution training set, and it measures both aspect accuracy and perplexity. Averaged over the four datasets, CTRL reports $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$13, $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$14, and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$15 with a gap of $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$16; Con.Prefix reports $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$17, $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$18, and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$19 with a gap of $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$20; DCG reports $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$21, $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$22, and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$23 with a gap of $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$24; Dis-Lens reports a gap of $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$25; and Prior and LLaMA-2 are summarized as having gaps around $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$26 and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$27, respectively. Meta-MCTG introduces an inner update on seen combinations and an outer update that penalizes pseudo-compositional recombinations, with $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$28 and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$29, improving compositional accuracy in $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$30 of cases and by at most $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$31, while perplexity remains essentially unchanged within approximately $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$32 point (Zhong et al., 2024).

A related result comes from the in-context learning literature. In a modular multitask setting designed to control compositional structure in the data generation process, transformers are reported to struggle to generalize compositionally in-context despite being expressive enough to do so. The abstracted conclusion is that compositional generalization becomes possible only when a bottleneck is introduced that enforces explicit separation between task inference and task execution (Kobayashi et al., 2024).

Taken together, these studies challenge the assumption that ordinary multitask exposure is sufficient for compositional transfer. They instead indicate that explicit structure—real multi-task dialogue data, meta-learned pseudo-compositional training, or an architectural bottleneck separating task inference from task execution—plays a decisive role.

6. On-device compositional multi-tasking and efficient adapter composition

The on-device LLM setting makes the definition of compositional multi-tasking operationally strict: a single test example requires multiple tasks to be applied sequentially in one inference pass. The benchmark introduced for this purpose contains four compositional tasks: Summarization + Translation, Summarization + Tone Adjustment, Reply Suggestion + Translation, and Reply Suggestion + Tone Adjustment. The main tasks are summarization or reply suggestion; the auxiliary tasks are translation or tone adjustment. Summarization tasks use DialogSum, reply tasks use Synthetic Persona Chat, translated outputs are created with OpusMT, and tone-adjusted outputs use a fine-tuned RedPajama-INCITE-Base 3B model. Prompts include forms such as “Summarize the following text and translate it from English to Spanish” and “Suggest a reply … and change its tone to {tone}” (Bohdal et al., 21 Jul 2025).

The benchmark keeps the example counts of the underlying main tasks: Summarization has $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$33 train/validation/test, Reply Suggestion has $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$34, Translation as a single task has $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$35, and Tone Adjustment as a single task has $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$36. Evaluation uses ROUGE-L for summarization-based tasks, Weighted ROUGE for reply-based tasks with

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$37

and an LLM Judge, defined as a binary $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$38 score from a 70B Llama model with task-specific prompts (Bohdal et al., 21 Jul 2025).

Methodologically, the paper compares prompting, single-task LoRA usage, in-context learning, multi-step LoRA usage, a joint-expert LoRA, and several merging baselines. Learnable Calibration begins from a linear merge

$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$39

and calibrates it on compositional data. The first variant learns a column-wise bias vector $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$40, giving approximately $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$41K parameters; the second learns an additional low-rank LoRA $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$42, giving approximately $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$43K parameters. Both preserve single-pass inference, and calibration parameters are shared across layers but separate per projection (Bohdal et al., 21 Jul 2025).

Method	# Inferences	Additional Storage (MB)
Multi-step LoRA usage	2×	0.00
Joint-expert LoRA	1×	57.10
Learnable Calibration	1×	0.05
Learnable Calibration++	1×	0.32

The main results are averaged across LLaMA 3.2 1B, Qwen2.5 1.5B, and StableLM2 1.6B, and across languages or tones per task. Fast merges and prompting are summarized as failing to solve both tasks, with LLM-J $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$44 for some settings. Multi-step LoRA usage and joint-expert LoRA achieve higher judged success but incur time or storage costs. Learnable Calibration reaches $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$45 on Summarization + Translation and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$46 on Reply Suggestion + Translation, while Learnable Calibration++ reaches $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$47, $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$48, $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$49, and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$50 across the four benchmark tasks. The paper summarizes these as $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$51–$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$52 LLM-J on Summarization + Translation and $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$53–$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$54 on Reply Suggestion + Translation, comparable to multi-step methods, with only a single pass and less than $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$55 MB extra storage. An ablation removing initialization from the single-task LoRAs degrades performance by $$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$56–$$T \in \{\text{QA}, \text{Summarization}, \text{Fact-checking}, \text{Cell-generation}\}$$57 points, indicating that calibrating existing adapters is critical (Bohdal et al., 21 Jul 2025).

This practical setting also clarifies a recurrent misconception. Multi-task support in the sense of “one-task-per-example” model merging is not the same as compositional multi-tasking. The latter requires that one prompt specify a composition and that the model satisfy the full sequence of operations in a single output. A plausible implication is that practical deployment depends not only on reusable single-task modules but also on explicit mechanisms for calibrating or structuring their interaction on compositional data.