Progressive Task-Specific Multi-Task Adaptation

• The paper demonstrates that sharing early layers while progressively specializing later ones using adapters and dynamic routing improves both accuracy and parameter efficiency.
• It details methods such as TGLoRA, task-specific normalization, and MoE-LoRA, showcasing how lightweight task-specific modulation balances shared and dedicated features.
• Empirical results across dense prediction and robotics domains confirm that progressive specialization mitigates task interference and enhances overall system performance.

Progressive task-specific multi-task adaptation denotes a family of multi-task learning strategies in which broad sharing is retained in early or task-agnostic components, while specialization is introduced progressively in later layers, grouped adapters, conditioned prompts, or dynamically routed submodules. In its explicit parameter-efficient formulation, adapter modules are shared across all tasks in the initial layers and become progressively more task-specific in the later layers (Gangwar et al., 23 Sep 2025). Closely related formulations appear in adaptive layer selection for fine-tuning (Wallingford et al., 2022), task-conditioned visual adapters for a single multi-task policy (Marza et al., 2024), task-specific normalization with soft capacity allocation (Suteu et al., 23 Dec 2025), prototype-based knowledge retrieval under partial supervision (Oh et al., 12 Jan 2026), and MoE-LoRA systems that separate general-purpose and specialist pathways under multi-task domain adaptation (Yang et al., 12 Jan 2026). An earlier precursor in sequence tagging proposed a neural network framework that supports domain adaptation for multiple tasks simultaneously, and learns shared representations that better generalize for domain adaptation (Peng et al., 2016).

1. Definition and scope

In the most direct usage, progressive task-specific multi-task adaptation is a parameter-efficient strategy for adapting a single pre-trained backbone to multiple downstream tasks by using lightweight task-grouped or task-specific modules whose sharing pattern changes with depth (Gangwar et al., 23 Sep 2025). The defining intuition is that early layers capture low-level or domain-general structure and therefore benefit from being shared, whereas later layers are closer to task decoders and therefore benefit from increasing specialization. A closely aligned formulation appears in task-adaptive parameter sharing, where a base model is kept frozen and each task learns perturbations only for a small, task-specific subset of layers (Wallingford et al., 2022).

Related work broadens the scope of the concept beyond parameter-efficient fine-tuning. A unified semantic communication system uses domain adaptation in the training procedure to specify the task-specific features for each task, so that the system only needs to transmit the task-specific features, rather than all the features, while a multi-exit architecture assigns different depths to different tasks (Zhang et al., 2022). In multi-task policy learning, task-conditioned adapters alter visual features inside a frozen pre-trained ViT so that a single policy can address multiple tasks, and unseen tasks can be handled by optimizing only a task embedding from a few demonstrations (Marza et al., 2024). This suggests that the term is best understood as an architectural and optimization principle rather than as a single model family.

2. Shared-early, specialized-late architectures

The clearest architectural realization is the task-grouped LoRA tree. In this design, TGLoRA inserts low-rank branches into attention and MLP layers, with one group in the earliest stage and progressively more groups in deeper stages. For each group $\pi$, the layer update is

$$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$

and the number of groups per layer is non-decreasing with depth, while groups can split across layers but never merge (Gangwar et al., 23 Sep 2025). This yields a tree-structured multi-task network in which each task path traverses shared adapters first and progressively more specific adapters later.

A more general layer-selection view is given by Task Adaptive Parameter Sharing. There, each task-specific layer is written as

$w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$

where $\bar{w}_i$ is the frozen base weight, $\delta w_i$ is the task-specific perturbation, and $I_\tau(s_i)$ activates specialization only when the learned score $s_i$ crosses a threshold (Wallingford et al., 2022). The corresponding optimization adds a sparsity penalty on the number of active layers: $$(\delta w, s) = \arg\min_{\delta w, s}\; \mathcal{L}_D(w) + \frac{\lambda}{L}\sum_{i=1}^{L} |s_i|.$$ This formulation makes progressive specialization literal: all layers begin shared, and only the subset that improves task loss sufficiently becomes task-specific.

A minimalist alternative argues that elaborate routing may be unnecessary. Task-Specific Sigmoid Batch Normalization keeps feature extractors fully shared and moves adaptation into normalization gates: $$TS\sigma BN(x; \gamma_t)_c = \sigma(\gamma_{t,c})\, \hat{x}_{t,c}, \qquad \hat{x}_{t,c} = \frac{x_c - \mu_{B,t,c}}{\sqrt{\sigma^2_{B,t,c} + \epsilon}}.$$ The learned task-filter importance matrix $I_{t,i} = \sigma(\gamma_{t,i})$ acts as a soft static gating mechanism per task and per filter, so early features can remain generic while deeper filters become increasingly specialized (Suteu et al., 23 Dec 2025). This suggests that progressive task specificity can emerge from lightweight per-task modulation rather than from explicit branches.

3. Task conditioning, retrieval, and expert routing

Several strands of work replace explicit layer branching with task-conditioned feature modulation. In multi-task policy learning, a frozen VC-1 ViT is augmented with middle adapters after each transformer layer and a top adapter after token aggregation. The modified layer is

$$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$0

and the final policy is conditioned on both adapted visual features and the task embedding $$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$1 (Marza et al., 2024). Because the task embedding is also optimized for unseen tasks from demonstration likelihood, this architecture implements a continuous form of task-specific adaptation rather than a discrete per-task head.

Prototype-based retrieval pushes the same idea toward partially annotated multi-task learning. A learnable prototype set

$$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$2

stores task-specific characteristics, while cosine-similarity affinities

$$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$3

drive a knowledge retrieval transformer that refines each task representation using prototype-weighted associations (Oh et al., 12 Jan 2026). The Association Knowledge Generating loss

$$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$4

stabilizes these task-specific characteristics without relying on predictions for unlabeled tasks.

Expert-routing formulations make specialization even more explicit. TaskExpert decomposes backbone features into $$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$5 expert features $$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$6 and assembles task-specific features via dynamic task-specific gating: $$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$7 The memory term $$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$8 is updated progressively across layers and acts as an additional feature expert (Ye et al., 2023). Med-MoE-LoRA applies the same principle to LLMs by replacing a single low-rank update with an MoE over LoRA experts,

$$y_\pi = W x_\pi + \gamma_{r,\pi} B_\pi A_\pi x_\pi, \qquad \gamma_{r,\pi} = \frac{\alpha_\pi}{r_\pi},$$9

and further partitions experts into an anchor pathway for preserving general-purpose reasoning and a specialist pathway for domain-specific gradients (Yang et al., 12 Jan 2026). A plausible implication is that progressive task-specific adaptation is compatible with both static parameter grouping and dynamic input-conditional routing.

4. Optimization and training regimes

One recurring design choice is to learn task relations explicitly and let those relations determine how much sharing each layer receives. In TGLoRA, task similarity is estimated from gradients with respect to the shared backbone: $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$0 and the similarity between tasks $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$1 and $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$2 is

$w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$3

Tasks are then partitioned into groups that maximize a partition score $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$4, and those groups are used to define the progressive branching pattern across stages (Gangwar et al., 23 Sep 2025). This makes “progressive” depend not only on depth but also on a data-derived task similarity structure.

A second regime is explicit stage-wise adaptation. In the pretrain-adapt-finetune paradigm for unified autonomous driving, off-the-shelf pretrained models are first kept intact, then a multi-scale adapter is trained while the backbone and task heads are frozen, and only afterward is the full model jointly fine-tuned (Liang et al., 2022). The adapt stage uses

$w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$5

with gradients active only for the adapter. This suggests a practical distinction between progressive specialization through architectural depth and progressive specialization through optimization stages.

A third regime uses task-aware dynamic modulation within parameter-efficient tuning. TADFormer inserts LoRA-style shared modules and task-adapting transformer blocks, derives task attention maps from prompts, and applies a Dynamic Task Filter: $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$6 The adapters remain lightweight, but the residual branch becomes input- and task-conditioned (Baek et al., 8 Jan 2025). In sequential dependence settings such as multi-step conversion estimation, progressive task coupling can also be enforced at the objective level by modeling adjacent tasks jointly and adding dependence losses between $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$7 and $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$8 (Tao et al., 2023). Taken together, these works indicate that progressive task-specific adaptation can be imposed through topology, conditioning, or optimization.

5. Empirical evidence across domains

Dense prediction provides the most direct quantitative support. On PASCAL-Context, progressive TGLoRA with attention, MLP, and “others” achieves $w_i = \bar{w}_i + I_\tau(s_i)\,\delta w_i,$9, compared with $\bar{w}_i$0 for MTLoRA and $\bar{w}_i$1 for a multi-task full fine-tuning baseline; on NYUD-v2 the same configuration reaches $\bar{w}_i$2, whereas MTLoRA gives $\bar{w}_i$3 and multi-task full fine-tuning gives $\bar{w}_i$4, while using roughly one-fifth of the trainable parameters required by multi-task full fine-tuning (Gangwar et al., 23 Sep 2025). TADFormer reports that the proposed method achieves higher accuracy in dense scene understanding tasks, while reducing the number of trainable parameters by up to 8.4 times when compared to full fine-tuning of MTL models; at rank $\bar{w}_i$5, TADFormer uses 4.78M trainable parameters and achieves $\bar{w}_i$6, whereas MTLoRA at the same rank uses 6.08M and achieves $\bar{w}_i$7 (Baek et al., 8 Jan 2025).

Dynamic task-specific representation learning shows the same pattern. TaskExpert improves PASCAL-Context from a baseline $\bar{w}_i$8 to $\bar{w}_i$9 when memory and context-aware gating are included, and on NYUD-v2 improves semantic segmentation from 53.56 to 55.35 mIoU while reducing surface-normal mean error from 19.04 to 18.54 (Ye et al., 2023). Task-specific normalization provides a contrastingly simple route: on NYUv2 with SegNet, hard parameter sharing achieves $\delta w_i$0, MTAN $\delta w_i$1, TSBN $\delta w_i$2, and TS$\delta w_i$3BN $\delta w_i$4 while using $\delta w_i$5 the parameters of single-task learning; on CelebA with 40 tasks, TS$\delta w_i$6BN reaches F1 $\delta w_i$7 at $\delta w_i$8 the single-task parameter cost (Suteu et al., 23 Dec 2025).

LLM adaptation adds evidence from stability–plasticity trade-offs. Med-MoE-LoRA reaches an average medical score of $\delta w_i$9, compared with $I_\tau(s_i)$0 for Vanilla MoE-LoRA, $I_\tau(s_i)$1 for Multi-LoRA, and $I_\tau(s_i)$2 for Standard LoRA; it also preserves general-domain capabilities with MMLU drop $I_\tau(s_i)$3 and GSM8K drop $I_\tau(s_i)$4, far smaller than the drops observed for full fine-tuning and standard LoRA (Yang et al., 12 Jan 2026). In robotics, task-conditioned visual adapters show that a single multi-task policy with non-adapted VC-1 features underperforms, while conditioned middle and top adapters make the multi-task policy competitive with and, on average, better than single-task VC-1 policies, and unseen tasks can be addressed by optimizing only the task embedding from a few demonstrations (Marza et al., 2024). This suggests that the empirical case for progressive task-specific adaptation is strongest when tasks are heterogeneous enough that fully shared features are insufficient, but still related enough to benefit from a common backbone.

6. Limitations, debates, and open directions

A central debate concerns how much architectural complexity is actually necessary. MoE routing, expert memories, and prototype transformers all deliver strong results, but Task-Specific Sigmoid Batch Normalization shows that normalization layers alone are sufficient to address many of these challenges and that complex MTL architectures may be unnecessary (Suteu et al., 23 Dec 2025). Conversely, TGLoRA and Med-MoE-LoRA argue that task interference and negative transfer are sufficiently severe in parameter-efficient multi-task learning that explicit grouping, routing, or expert partitioning is beneficial (Gangwar et al., 23 Sep 2025, Yang et al., 12 Jan 2026). A plausible implication is that the right degree of specialization is domain-dependent: some regimes reward structural simplicity, whereas others reward explicit task decoupling.

The literature also identifies recurrent failure modes. Static expert topology limits adaptivity when new task distributions appear; soft, adaptive routing can suffer from expert underuse or collapse; effective multi-task adaptation requires sufficiently rich and diverse datasets across sub-tasks; and exact grouping procedures become difficult to scale as the number of tasks grows (Yang et al., 12 Jan 2026, Gangwar et al., 23 Sep 2025). Prototype-based methods reduce dependence on pseudo-labels, but still rely on stable prototype learning and well-parameterized association spaces (Oh et al., 12 Jan 2026). From a broader perspective, these limitations suggest that progressive task-specific multi-task adaptation remains a design space rather than a settled recipe. The common direction, however, is consistent: preserve a strong shared substrate, introduce specialization gradually where interference accumulates, and let task relations—whether learned from gradients, prompts, prototypes, or gates—govern how that specialization emerges.