TUNA: Task-Specific and Universal Adapters

• TUNA is a paradigm that decomposes models into a shared universal adapter and lightweight task-specific modules to enable efficient transfer across tasks.
• It employs entropy-based selection and max-out adapter fusion to combine specialized predictions with universal knowledge, reducing inference errors.
• Empirical results on CIFAR100, ImageNet-R, ImageNet-A, and ObjectNet benchmarks show TUNA outperforms prior methods without relying on exemplar storage.

Searching arXiv for the specified TUNA paper and closely related adapter-based works. Task-Specific and Universal Adapters (TUNA) denotes a line of adapter-based model design in which task-specialized modules are combined with a shared or universal component, typically to reconcile specialization with transfer across tasks, domains, or incremental stages. In its most explicit recent formulation for pre-trained model-based class-incremental learning (CIL), TUNA integrates per-task adapters with a fused universal adapter on top of a frozen Vision Transformer, then combines entropy-based task-adapter selection with universal knowledge at inference time (Wang et al., 11 Aug 2025). Closely related uses of the same task-specific/universal decomposition appear in universal representation learning, lifelong learning, parameter-efficient fine-tuning, and preference-aware multi-task learning, although the exact training protocol, inference assumptions, and role of the universal component differ substantially across works (Li et al., 2022).

1. Scope of the TUNA paradigm

The central TUNA motif is a decomposition between a shared component that captures broadly reusable structure and lightweight task-specific modules that absorb specialized variation. In different papers, the shared part may be a frozen pre-trained backbone, a continually updated backbone, a universal encoder distilled from specialists, or base weights augmented by task-specific low-rank residuals. The task-specific part may be an adapter inserted into transformer MLP blocks, a $1\times 1$ convolutional adapter attached to a universal encoder, or a low-rank residual module (Wang et al., 11 Aug 2025).

Work	Setting	Universal / task-specific decomposition
"Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning" (Wang et al., 11 Aug 2025)	Class-incremental learning	Frozen ViT + task adapters + fused universal adapter
"Universal Representations: A Unified Look at Multiple Task and Domain Learning" (Li et al., 2022)	Multi-task and multi-domain learning	Universal encoder + task/domain adapters + task heads
"Lifelong Learning with Task-Specific Adaptation: Addressing the Stability-Plasticity Dilemma" (Wang et al., 8 Mar 2025)	Lifelong learning / Task-IL	Universal backbone + task adapters + task heads
"Swin-TUNA : A Novel PEFT Approach for Accurate Food Image Segmentation" (Chen et al., 23 Jul 2025)	PEFT for segmentation	Frozen Swin backbone + multiscale trainable adapters + learned gates
"Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences" (Dimitriadis et al., 2024)	Preference-aware multi-task learning	Shared base weights + task-specific low-rank adapters

This suggests that TUNA is best understood not as a single invariant algorithm, but as a family of architectures that separate shared representation capacity from task-local adaptation capacity. What varies across instantiations is how the universal component is obtained, whether the backbone is frozen or co-trained, how task identity is inferred or supplied, and whether the final prediction is produced by selection, fusion, distillation, or convex combination.

2. TUNA in pre-trained model-based class-incremental learning

In the CIL formulation, the backbone is a frozen pre-trained Vision Transformer with $L$ transformer blocks. For each incremental task $t=1,\dots,T$, a lightweight task-specific adapter is inserted into every transformer block’s MLP sublayer via a residual connection, while all pre-trained weights remain fixed and only adapter parameters and task classifiers are trained (Wang et al., 11 Aug 2025).

For layer $\ell$, with $h^{(\ell-1)}$ denoting the input to the MLP sublayer, the task-specific adapter $A_t$ is a two-layer bottleneck:

$$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$

where $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$ is the down-projection, $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$ is the up-projection, $\sigma(\cdot)$ is ReLU, $L$0 is the ViT feature dimension, and $L$1 is the adapter bottleneck. In each block, the MLP output becomes

$L$2

The distinctive addition is a universal adapter $L$3, constructed only after learning the full set of task adapters $L$4. It has the same insertion points and functional form as the task adapters, but its parameters are obtained by adapter fusion rather than direct supervised training (Wang et al., 11 Aug 2025).

The motivation is explicit: existing pre-trained model-based CIL methods often freeze the pre-trained network and adapt to incremental tasks using additional lightweight modules such as adapters, but incorrect module selection during inference hurts performance, and task-specific modules often overlook shared general knowledge, leading to errors on distinguishing between similar classes across tasks. TUNA addresses these two issues jointly by combining a task-specialized path with a universal path.

3. Selection, fusion, and learning objectives

At test time, for an input $L$5, TUNA runs $L$6 forward passes: one through each task adapter $L$7 and one through the universal adapter $L$8. For each task adapter, the class-probability vector is

$L$9

and uncertainty is measured by Shannon entropy

$t=1,\dots,T$0

The selected task adapter is the minimum-entropy one,

$t=1,\dots,T$1

This entropy-based selection mechanism is the method’s answer to the absence of task labels at class-incremental inference time (Wang et al., 11 Aug 2025).

The universal adapter is produced by fusing learned task adapters. Each adapter’s parameters are flattened into a vector $t=1,\dots,T$2. For each coordinate, fusion first performs sign voting,

$t=1,\dots,T$3

then applies magnitude pooling by selecting the largest-absolute-value element among the $t=1,\dots,T$4 that agrees with the voted sign, and finally recombines sign and magnitude to obtain $t=1,\dots,T$5. The fused vector is reshaped into the universal adapter parameter tensors. The paper characterizes this as preserving dominant directions and maximal magnitudes in a “max-out” style across tasks (Wang et al., 11 Aug 2025).

The final prediction combines the selected task adapter and the universal adapter. With logits $t=1,\dots,T$6 and $t=1,\dots,T$7, the predicted label is

$t=1,\dots,T$8

An equivalent probability-level formulation is

$t=1,\dots,T$9

with $\ell$0 or tuned on a held-out set, although simple score summation worked well in the paper (Wang et al., 11 Aug 2025).

Training for task $\ell$1 introduces a fresh adapter $\ell$2 and new classifier weights for classes in $\ell$3. The objective is

$\ell$4

where $\ell$5 is cross-entropy over $\ell$6, and the orthogonality regularizer decorrelates the current adapter’s up-projection weights from those of previous adapters:

$\ell$7

There is no direct supervised training on $\ell$8; it is entirely constructed via fusion after the $\ell$9 tasks (Wang et al., 11 Aug 2025).

4. Empirical results in class-incremental learning

The reported evaluation uses CIFAR100 with 100 classes and $h^{(\ell-1)}$0 $h^{(\ell-1)}$1, ImageNet-R with 200 classes and $h^{(\ell-1)}$2 $h^{(\ell-1)}$3, ImageNet-A with 200 classes and $h^{(\ell-1)}$4 $h^{(\ell-1)}$5, and ObjectNet with 200 classes and $h^{(\ell-1)}$6 $h^{(\ell-1)}$7. The backbone is ViT-B/16 pretrained on IN-21K and IN-1K. Baselines span prompt-based methods such as L2P, DualPrompt, and CODA-Prompt; full-fine-tuning via SLCA; adapter-tuning methods including SSIAT, MOS, and EASE; prototype-based SimpleCIL; and first-session adapters such as APER and RanPAC (Wang et al., 11 Aug 2025).

TUNA is reported to outperform all prior methods on these benchmarks. The average accuracies given are $h^{(\ell-1)}$8 on CIFAR $h^{(\ell-1)}$9 $A_t$0, compared with $A_t$1 for the best prior method; $A_t$2 on ImageNet-R $A_t$3 $A_t$4, compared with $A_t$5; $A_t$6 on ImageNet-A $A_t$7 $A_t$8, compared with $A_t$9; and $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$0 on ObjectNet $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$1 $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$2, compared with $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$3 (Wang et al., 11 Aug 2025).

The method is also positioned against exemplar-based CIL. Under the same pre-trained ViT backbone, TUNA with $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$4 exemplars is reported to beat classical exemplar-based CIL methods including iCaRL, DER, FOSTER, MEMO, and TagFex. This is a substantive point because the approach does not store exemplars, yet remains competitive or superior in a regime where rehearsal is a standard baseline (Wang et al., 11 Aug 2025).

Ablation results isolate the contribution of the three main components. Starting from a baseline with no entropy selection, no orthogonality regularization, and no universal adapter, entropy selection contributes approximately $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$5 to $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$6, orthogonality contributes $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$7 to $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$8, and the universal adapter contributes $$\mathbf{h}_{t}^{(\ell)} = W_{t,2}^{(\ell)}\,\sigma\bigl(W_{t,1}^{(\ell)}\,\mathbf{h}^{(\ell-1)} + b_{t,1}^{(\ell)}\bigr) + b_{t,2}^{(\ell)},$$9 to $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$0. The paper’s summary is that TUNA disentangles task-specific expertise through orthogonal adapters, captures shared cross-task knowledge through max-out fusion, and combines the two at inference through entropy-driven selection and ensemble, thereby achieving state-of-the-art class-incremental performance without storing exemplars (Wang et al., 11 Aug 2025).

5. Related formulations in universal representation learning, lifelong learning, and PEFT

A precursor to the task-specific/universal adapter theme appears in universal representation learning, where a single universal encoder $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$1 is paired with lightweight task- or domain-specific adapters $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$2 and heads $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$3. Training minimizes a combination of task supervision, feature distillation, and prediction distillation against specialist teacher networks. In this setting, the adapters project universal features into each specialist’s representation space, and the resulting universal model achieves state-of-the-art performance on dense prediction in NYU-v2 and Cityscapes, multi-domain classification on Visual Decathlon, and cross-domain few-shot learning on MetaDataset. Reported values include NYU-v2 performance of $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$4 mIoU, $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$5 depth error, and $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$6 normal error for TUNA, and a Visual Decathlon average accuracy of $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$7 with decathlon score $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$8 at $W_{t,1}^{(\ell)}\in\mathbb{R}^{r\times d}$9params $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$0 specialist (Li et al., 2022).

In lifelong learning, AdaLL adopts a two-block architecture consisting of a universal backbone network $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$1 and task-specific adapters $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$2, with the explicit goal of addressing the stability-plasticity dilemma. Here, unlike the frozen-backbone CIL TUNA, the backbone is co-trained under regularization constraints, while the adapters remain task-specific and unconstrained. Stability is attributed to prediction regularization and EWC-style weight regularization on the backbone, whereas plasticity is attributed to unconstrained adapters and classifiers plus soft backbone adaptation. The paper reports that co-training $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$3 outperforms freezing $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$4 and updating only $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$5, and it notes that inference in Task-IL activates only $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$6 and $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$7, with a task-ID oracle choosing the correct pair (Wang et al., 8 Mar 2025).

Swin-TUNA instantiates the same broad division between frozen universal features and trainable task-specific adaptation in a PEFT setting for food image segmentation. Built on a four-stage Swin Transformer, it inserts multiscale trainable adapters in parallel with the original MLP at each stage. Each adapter contains a down-projection, depth-wise separable convolution, up-projection, GeLU, Dropout, and a learned scalar gate; fusion between the frozen MLP output and the adapter output is controlled by learned scalars $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$8. Only $W_{t,2}^{(\ell)}\in\mathbb{R}^{d\times r}$9M parameters are trainable, approximately $\sigma(\cdot)$0 of the model, compared with $\sigma(\cdot)$1M for full fine-tuning in FoodSAM, and the method reports $\sigma(\cdot)$2 mIoU on FoodSeg103 and $\sigma(\cdot)$3 on UECFoodPix Complete, while reducing the parameter count by $\sigma(\cdot)$4 (Chen et al., 23 Jul 2025).

PaLoRA is not named TUNA, but it embodies a closely related architecture-level principle: shared base parameters $\sigma(\cdot)$5 capture general features, while task-specific low-rank adapters $\sigma(\cdot)$6 capture specialized corrections. The attainable residuals form the convex hull of the task-specific modules, yielding a preference-parameterized Pareto subspace $\sigma(\cdot)$7. The method reports memory overhead reductions of $\sigma(\cdot)$8 to $\sigma(\cdot)$9 relative to competing Pareto front learning baselines, with parameter overhead fractions of about $L$00 to $L$01 in Cityscapes and NYU experiments (Dimitriadis et al., 2024).

Taken together, these formulations show that the universal/task-specific split is portable across distinct problem settings. What changes is whether the universal component is frozen, distilled, co-trained, or queried under a preference vector, and whether specialization is retrieved by entropy, by an oracle task ID, by explicit heads, or by convex combination.

6. Limitations, misconceptions, and open directions

A common misconception is that TUNA names a single standardized architecture. The literature instead contains multiple incompatible instantiations: a fused universal adapter over a frozen ViT for Class-IL, a distilled universal encoder with per-task adapters for multi-task and multi-domain learning, a co-trained universal backbone for lifelong learning, and a PEFT segmentation module with learned universal/task-specific gating (Wang et al., 11 Aug 2025).

Another misconception is that adding task-specific adapters automatically resolves inference-time ambiguity. In the CIL formulation, incorrect module selection during inference is identified as a direct source of performance degradation, which is why entropy-based selection is introduced. In AdaLL’s Task-IL formulation, the problem is sidestepped by a task-ID oracle; the same paper explicitly notes that Class-IL is harder and that preliminary appendix results show smaller gains (Wang et al., 8 Mar 2025).

The limitations are similarly setting-dependent. In the universal-representation formulation, two-stage training requires pre-training one teacher per task or domain, memory and compute cost scale with the number of specialists during distillation, and performance depends on teacher quality and diversity (Li et al., 2022). In AdaLL, stored adapters still add overhead and merging many tasks may become costly if $L$02 is very large (Wang et al., 8 Mar 2025). In the CIL TUNA formulation, the universal adapter is not directly supervised but instead constructed entirely via fusion, so its quality depends on the learned geometry of the task adapters (Wang et al., 11 Aug 2025).

Open directions are stated explicitly in the lifelong-learning work: applying adapter fusion when tasks are related; adapting to Class-IL where task-ID is unavailable, perhaps via multi-head soft gating or task-inference modules; extending the framework to non-vision domains such as continual RL or speech and to very large backbones; and developing theoretical analyses of the stability-plasticity trade-off under adapter regularization (Wang et al., 8 Mar 2025). A plausible implication is that future TUNA-style methods will be differentiated less by the existence of task-specific and universal components than by how they infer, construct, and calibrate the interaction between them under realistic deployment constraints.