---
title: 'MTIL: Multi-Domain Task Incremental Learning'
url: https://www.emergentmind.com/topics/multi-domain-task-incremental-learning-mtil
type: topic
---

# MTIL: Multi-Domain Task Incremental Learning

Multi-domain Task Incremental Learning (MTIL) is a continual-learning regime in which tasks arrive sequentially from multiple distinct data domains, and a model must assimilate each new task while retaining competence on previously seen domains under restricted access to past data. In representative formulations, a task sequence \(T_1,\dots,T_t\) is associated with per-task training and test sets, only the current task’s training data are accessible at stage \(t\), and the objective is sustained performance over the union of seen test distributions [2501.08878]. Across the literature, however, MTIL is not a single protocol but a family of related settings spanning shared-label domain-incremental learning, disjoint-class task streams, task-aware and task-agnostic inference, and frozen-backbone vision-language adaptation [2410.00911][2605.25708][2508.16463].

## 1. Problem definitions and protocol variants

A canonical MTIL formulation assumes a sequence of tasks \(T_1,\dots,T_t\) drawn from different data domains, with task \(T_i\) associated with a training set \(D^s_i\) and a test set \(D^t_i\), where each \(D^s_i\subset\mathcal X\times\mathcal Y\). Domains can shift arbitrarily, and the stream may collect subtasks such as class-increments across multiple domains. At time \(t\), the learner sees only \(D^s_t\), with all previous data unavailable, and seeks a predictor that performs well on \(\cup_{i=1}^t D^t_i\) under a cross-entropy objective [2501.08878]. A closely related parameterization appears in factorized-tensor formulations, where a fixed backbone with shared weights \(\mathcal W_{\rm shared}\) is augmented per task by \(\Delta\mathcal W_t\), so that \(\hat y_t=\mathbf F(x;\mathcal W_{\rm shared}+\Delta\mathcal W_t,h_t)\) [2310.06124].

The term also subsumes narrower special cases. In Domain-Incremental Learning (DIL), the label space \(Y\) is fixed across all domains, \(p_b(X)\neq p_{b'}(X)\), no past examples are stored, and the goal is a single classifier that predicts over the shared label set without catastrophic forgetting. This is explicitly mapped as \( \mathrm{DIL}\subset \mathrm{MTIL}\), with tasks identified with domains and no task identity at inference [2410.00911]. By contrast, some MTIL works assume disjoint class sets \(\mathcal Y_t\cap\mathcal Y_{t'}=\emptyset\), requiring both task routing and within-task classification when task identity is absent [2605.25708][2506.19608].

Inference assumptions vary materially across the literature.

| Representative setting | Inference assumption | Label structure |
|---|---|---|
| MSDEM | performance on \(\cup_{i=1}^t D^t_i\) after seeing only \(D^s_t\) | subtasks across multiple domains |
| Duct | no task identity at inference | fixed label space \(Y\) |
| ITL-IRU | domain-ID known at test time | each domain is a task |
| MoDER / semantic segmentation | current domain or task known at inference | domain-specific classes or label spaces |
| CMAP / AFA / ChordPrompt | no task ID given at inference | disjoint class sets across tasks |

This heterogeneity is central to MTIL. A common misconception is to treat all reported results as directly comparable; the published protocols differ in whether label spaces are shared, whether task identity is available at test time, and whether “future-task” zero-shot transfer is part of the objective [2207.09074][2508.16463][2110.12205].

## 2. Architectural families

A dominant architectural pattern is the use of a frozen or largely frozen backbone with small task- or domain-specific augmentations. FTN freezes a backbone \(\mathcal W_{\rm shared}=\{\mathbf W_1,\dots,\mathbf W_L\}\) and adds low-rank tensor corrections \(\Delta\mathbf W_{\ell,t}\) plus task-specific batch-normalization parameters and heads; each convolutional layer is updated by
\[
\mathbf W_{\ell,t}=\mathbf W_\ell+\Delta \mathbf W_{\ell,t},\qquad
\Delta \mathbf W_{\ell,t}=\sum_{r=1}^R \mathbf w^{(r)}_{1,\ell,t}\otimes \mathbf w^{(r)}_{2,\ell,t}\otimes \mathbf w^{(r)}_{3,\ell,t}.
\]
This preserves the source model while allocating only a small per-task factorization [2310.06124]. A closely related strategy appears in low-rank incremental updates, where each layer weight is expressed as a sum of frozen low-rank factors from prior tasks and a new rank-1 or low-rank increment for the current task [2207.09074].

Another long-running line uses reparameterization rather than full-copy expansion. In reparameterized-convolution models, a convolution is decomposed into a fixed filter bank \(B\) and a small task-specific modulator \(M^i\), with \(W^i\simeq M^i\cdot B\). The shared \(k\times k\) filters are frozen, and only the task-specific \(1\times 1\) modulators are trained, which the authors argue eliminates cross-task interference “by construction” [2007.12540]. Network-latent tensor factorization instead groups convolutional filters across layers into a higher-order tensor and reconstructs each domain’s weights through a shared Tucker core \(K\) and domain-specific factor matrices \(\{F_d^{(n)}\}\), thereby exploiting inter-layer correlations rather than treating layers independently [1904.06345].

Dynamic expansion is more explicit in expert-based systems. MSDEM uses multiple frozen Vision-Transformer backbones, fuses their class-token features, and instantiates a new expert module \(\mathcal E_t=\{f_{\xi_t},f_{\omega_t}\}\) for each new task, with all previous experts frozen once trained [2501.08878]. DE&E places a Mixture-of-Experts head over a frozen, domain-agnostic feature extractor \(F(\cdot)\) and maintains expert networks \(f_i(\cdot)\) together with key vectors \(k_i\) for differentiable soft-KNN gating across text, image, and audio inputs [2307.05399]. In semantic segmentation, the analogous decomposition is a Domain-Aware Residual Unit with universally shared \(3\times 3\) convolutions and per-domain adapters and BatchNorm parameters, so that approximately \(78.8\%\) of weights are shared and about \(21.2\%\) are domain-specific [2110.12205].

Recent vision-language MTIL methods keep CLIP frozen and specialize only prompts, gates, or lightweight adapters. ChordPrompt learns banks of visual and textual prompts and two cross-modal Aligner matrices \(A_{V2T}\) and \(A_{T2V}\), inserting prompts into both encoders at every layer [2506.19608]. CMAP introduces task-specific prompt pools and per-layer gating parameters on top of a frozen CLIP backbone \((f_v,f_t)\) with embedding dimension \(d=512\) [2605.25708]. AFA inserts a shared LoRA adapter into the final Transformer FFN of both branches and combines it with a Mixture-of-Experts adapter distributed through Transformer FFNs [2505.07690].

## 3. Routing, attention, and selective transfer

As MTIL shifted from task-aware to task-agnostic and zero-shot-aware protocols, routing became a central design problem. MSDEM addresses routing at two levels. First, its Dynamic Expandable Attention attaches a new multi-head attention block \((Q^t,K^t,V^t)\) for each task and computes
\[
\mathbf z^t_{\rm att}
= \mathrm{Softmax}\bigl(\hat Q^t(\hat K^t)^\top/\sqrt{d_k}\bigr)\hat V^t,
\]
so the current task can selectively weigh the relevance of multiple pre-trained backbones while earlier attention weights remain frozen [2501.08878]. Second, its Dynamic Graph Weight Router defines a learnable relation matrix \(C\in\mathbb R^{t\times t}\), uses a Gumbel-Softmax relaxation to produce a soft subset \(\hat M^t\) of prior experts, and forms \(\mathbf Z^t=\sum_{j=1}^t \bar{\mathbf z}^j\hat M^t[j]\), followed by a second attention block over the routed representation [2501.08878].

DE&E uses an explicitly differentiable gating mechanism rather than task-conditioned attention. For an embedding \(z=F(x)\), cosine distances to expert keys \(k_i\) are converted into a Gaussian-kernel matrix, followed by Sinkhorn/Bregman projection to solve an entropy-regularized optimal transport problem between a uniform row-marginal and a fixed column marginal. The second column of the transport plan yields gating weights \(y_i\in[0,1]\), which act like a soft indicator of whether expert \(i\) is among the \(K\) nearest keys to \(z\) [2307.05399]. This is combined with cosine similarity scores \(s_i=\cos(z,k_i)\) in the ensemble prediction.

Vision-language MTIL has largely reframed routing as prototype matching. CMAP’s “Text-Space Task Routing” forms one frozen text prototype per task,
\[
\boldsymbol\tau_t=\frac1{|\mathcal Y_t|}\sum_{c\in\mathcal Y_t}\mathbf e_c,
\]
and routes by maximizing \(\mathrm{sim}(f_v(\mathbf x),\boldsymbol\tau_t)\). The method supplements this with “Multi-Prototype Visual-Textual Confidence,” combining \(K=3\) K-means visual prototypes per class with text-alignment scores under task-adaptive confidence thresholds, and “Symmetric Cross-Modal Gating,” which applies Hard Gumbel gates to the text encoder conditioned on batch image features [2605.25708]. ChordPrompt similarly stores normalized domain prototypes
\[
P^i=\frac{\sum_{c\in\mathbb C_i}\hat Y^i_c}{\left\lVert\sum_c \hat Y^i_c\right\rVert}
\]
and retrieves the prompt bank of the prototype with maximal cosine similarity, falling back to vanilla zero-shot CLIP if the similarity does not exceed a threshold \(\gamma\) [2506.19608].

MoDER extends selective transfer from routing to recomposition. It stores a LoRA-based textual expert \(\tau_i\) for every seen class in a foundational hub, retrieves the top-\(K\) seen classes most similar to an unseen class in vanilla CLIP text space, and forges an update
\[
\tau_j=\sum_{\ell=1}^K w_{i_\ell}\tau_{i_\ell},\qquad
w_{i_\ell}=\mathrm{softmax}_\ell(\mathrm{sim}(i_\ell,j)),
\]
before applying an \(\alpha\)-smoothing step \(W_j=W+\alpha\tau_j\) to generate a refined prototype for the unseen class [2508.16463]. This suggests a broader MTIL trend: routing is increasingly used not only to preserve old competence but also to improve forward generalization to unseen domains.

## 4. Forgetting mitigation and optimization strategies

One major MTIL strategy is strict parameter isolation. In FTN, the original backbone is frozen after source training, previously learned factors remain frozen, and task \(t\) uses only its own \(\Delta\mathcal W_t\), batch-normalization parameters, and head. The paper states that, since there is no update to shared or earlier factors once they are learned, FTN “by construction entirely isolates parameter updates for different tasks, guaranteeing zero interference and thus no forgetting” [2310.06124]. ITL-IRU follows the same logic: frozen low-rank basis directions are never changed, and only the new low-rank update and selector matrices are learned for each incoming task, yielding “zero forgetting” in its reported benchmarks [2207.09074]. Reparameterized convolutions make the same claim in terms of task interference rather than low-rank updates: fixed shared filters plus frozen old modulators prevent gradient collision among tasks [2007.12540].

A second strategy is controlled expansion with frozen history. MSDEM updates only the new expert’s parameters \(\{\xi_t,\omega_t\}\), the new attention heads \(\{Q^t,K^t,V^t,Q^t_g,K^t_g,V^t_g\}\), and the new routing row \(C(t)\), while previous experts, attention blocks, and routing structure remain frozen [2501.08878]. AFA similarly freezes previously learned routers and decomposes learning into two stages: ABFA trains a fresh router with a Mixture-of-Experts adapter on the current task’s few-shot labels, and AFFA then updates only the shared forward adapter through a symmetric contrastive loss to accumulate task-invariant knowledge across tasks [2505.07690].

A third strategy is consolidation or distillation rather than strict isolation. Duct constructs a merged representation
\[
\phi^m=\phi_0+\alpha_\phi\sum_{i=1}^b \mathrm{Sim}_{0,i}\cdot \Delta\phi_i
\]
to consolidate domain-specific backbone updates, then retrains the new classifier on the merged embedding space and uses optimal transport to estimate compatible old-domain classifiers, merging them as
\[
W_o^m=(1-\alpha_W)W_o+\alpha_W W_nT.
\]
No replay or distillation losses are added beyond this explicit representation and classifier consolidation [2410.00911]. In semantic segmentation, forgetting is controlled through a knowledge-distillation retention term
\[
L_{KLD}=\lambda_{KLD}\sum_{i=1}^{t-1}\sum_{x\in D_t} D_{KL}(M_t(x;i)\|M_{t-1}(x;i)),
\]
combined with differential learning rates between shared and domain-specific parameters [2110.12205]. In object detection, Attentive Feature Distillation uses bottom-up self-attention and top-down box attention to preserve critical regions of feature maps during task transitions, and in difficult domain-plus-category shifts it is combined with exemplar sampling [2002.05347].

A fourth strategy uses replay without old labels. Kar et al. formulate continual expansion of an \(n\)-task NLP model into an \(n+1\)-task model through knowledge distillation on unlabeled data \(\mathcal U_i\) drawn from old-task distributions, optimizing
\[
L_{\rm total}=L_{\rm new}+\lambda L_{\rm KD}.
\]
With temperature \(T=2\) and \(\lambda=1\), this “UKD” procedure is reported to prevent “up to 20% performance drops on old tasks” and to nearly match multi-task retraining while never revisiting old labels [2302.11074]. MoDER uses a different replay substrate—synthetic CLIP image embeddings generated by a light diffusion model—to train per-class textual experts with a sigmoid-style alignment loss and preserve composability across tasks [2508.16463].

## 5. Benchmarks, metrics, and empirical landscape

Empirical MTIL spans several benchmark families rather than a single evaluation suite. MSDEM evaluates on TinyImageNet, CIFAR-100, CIFAR-10, and Birds, using two- and three-domain sequences and one four-domain chain, with “Average” defined as mean accuracy across all tasks and “Last” as accuracy on the final task [2501.08878]. CMAP, ChordPrompt, AFA, and MoDER all use an 11-dataset vision benchmark—Aircraft, Caltech101, CIFAR-100, DTD, EuroSAT, Flowers102, Food101, MNIST, OxfordPet, StanfordCars, and SUN397—totalling 1,201 classes, but metric conventions are not fully uniform. CMAP defines Transfer as forward-generalization to unseen tasks, Last as final retention on seen tasks, and Average as mean accuracy across all tasks [2605.25708]. MoDER also reports Transfer, Average, and Last, with task identity known at inference in its MTIL protocol [2508.16463]. AFA states that, in its MTIL reporting, “Average = (Transfer + Last)/2” [2505.07690]. Segmentation and detection works instead use mIoU, \(\Delta_m\%\), or mAP [2110.12205][2002.05347].

Within this heterogeneous landscape, several results are recurrently cited because they illustrate the design trade-offs of MTIL. On the MSDEM benchmark, “MSDEM-2 (two ViT backbones) outperforms all baselines (DER, DER++, DER+++refresh, MoE adapters, prompt-based StarPrompt, RanPac, Dap) by large margins in both Average and Last, often \(+15\%\!-\!20\%\) over the strongest prompt-replay methods.” The same study reports that MSDEM-2 uses approximately \(25\) M trained parameters versus approximately \(86\) M in StarPrompt-2nd stage, needs approximately \(1.7\) GiB GPU versus approximately \(11\) GiB, and runs approximately \(8.9\) it/s versus approximately \(1.2\) it/s [2501.08878].

FTN illustrates the parameter-efficiency extreme. On ImageNet→Sketch with ResNet-50, fine-tuning per domain uses 141 M convolutional parameters with mean top-1 \(85.98\), while FTN with \(R=50\) uses 36.02 M parameters with mean top-1 \(85.76\). On DomainNet with ResNet-34, fine-tuning per domain uses 127.7 M parameters with mean top-1 \(66.93\), while FTN with \(R=40\) uses 25.22 M parameters with mean top-1 \(66.29\). Across all listed benchmarks, FTN is reported to match or closely approach single-task performance while adding only a small fraction of the parameters used by full fine-tuning or prior adapter methods [2310.06124].

On the 11-dataset CLIP-based MTIL benchmark, several families have progressively improved forward transfer and final retention. ChordPrompt reports, under Order I, Transfer \(69.5\%\) versus ZSCL \(64.2\%\), Average \(75.6\%\) versus \(74.5\%\), and Last \(85.1\%\) versus \(83.4\%\), with approximately \(9.5\) M trainable parameters [2506.19608]. AFA reports Transfer \(70.3\%\), Average \(78.5\%\), and Last \(87.2\%\), while reducing trainable parameters from \(59.8\) M in BCL to \(30.1\) M and GPU memory from \(22.4\) GiB to \(18.1\) GiB [2505.07690]. MoDER reports Transfer \(69.7\), Average \(76.9\), and Last \(85.8\), and explicitly claims improvement over the zero-shot CLIP baseline on Transfer [2508.16463]. CMAP reports that text-space routing, multi-prototype visual-textual confidence, and symmetric gating improve Order-I performance over IAP, with ablations showing \(-\)text routing reduces Transfer by \(2.3\) percentage points, \(-\)MPVTC reduces Average by \(2.2\) percentage points, and \(-\)sym-gating reduces Average by \(2.9\) percentage points and Last by \(1.8\) percentage points [2605.25708].

Other subdomains reveal complementary empirical patterns. Duct reports on ViT-B/16-IN1K that Office-Home improves from Finetune \(\bar A=78.3\), \(A_B=76.2\) to Duct \(\bar A=86.3\), \(A_B=86.9\); DomainNet improves from L2P \(\bar A=50.5\), \(A_B=48.7\) to Duct \(\bar A=67.2\), \(A_B=67.0\); CORe50 improves from CODA-Prompt \(\bar A=87.9\), \(A_B=91.6\) to Duct \(\bar A=91.9\), \(A_B=94.5\) [2410.00911]. In semantic segmentation, the proposed domain-incremental method achieves CS \(65.21\) and BDD \(55.73\) on CS→BDD versus fine-tuning at CS \(40.05\) and BDD \(52.74\), with \(\Delta_m\%=3.55\%\) [2110.12205]. In object detection, Attentive Feature Distillation improves KITTI→Kitchen from 10.9 under fine-tuning to 36.6 without exemplars and 68.6 with 100 exemplars [2002.05347]. In NLP, UKD maintains MNLI at approximately \(83.3\) after a five-task chain, compared with approximately \(79.0\) under classical TKD [2302.11074].

## 6. Conceptual issues, misconceptions, and research directions

The first conceptual issue is that MTIL is not synonymous with a single inference model. Some methods require task or domain identity at test time, including ITL-IRU, the semantic-segmentation setting, and the MTIL protocol adopted by MoDER; others explicitly require no task identity and therefore must solve routing and confidence estimation as part of inference, including Duct, CMAP, ChordPrompt, and AFA [2207.09074][2110.12205][2508.16463][2410.00911]. A plausible implication is that “state of the art” claims should be interpreted within protocol families rather than across all MTIL papers indiscriminately.

The second issue concerns what actually causes forgetting. In object detection, the reported finding is that “domain gaps have smaller negative impact on incremental detection, while category differences are problematic,” especially in the fully multi-domain plus category-incremental scenario [2002.05347]. This contrasts with formulations where the label space is fixed and only the input distribution shifts, for which representation consolidation or per-domain adapters can be highly effective [2410.00911][2110.12205]. The literature therefore distinguishes at least two sources of difficulty: representation drift under domain shift, and classifier or routing ambiguity under class-set growth.

The third issue is the role of the frozen foundation model. Many methods rely on a strong pre-trained backbone—ImageNet-pretrained ResNets, ViT-B/16, or CLIP—and treat continual learning primarily as structured parameter allocation around that backbone. The papers themselves note the downside: FTN’s future directions include adaptive per-layer rank selection and shared evaluation among tasks [2310.06124]; Duct notes that merging weaker backbones may degrade and that optimal-transport cost grows with the number of classes and tasks [2410.00911]; DE&E remarks that the performance upper bound is tied to the quality of the frozen feature extractor and that the Sinkhorn-based gating is approximately \(3\times\) slower than non-differentiable E&E [2307.05399].

Current research directions largely extend the same themes. CMAP argues for exploiting CLIP’s text embedding space rather than relying exclusively on visual routing [2605.25708]. ChordPrompt proposes deeper cross-modal prompt exchange and domain-adaptive retrieval [2506.19608]. AFA identifies separate mechanisms for forward forgetting and backward forgetting, and notes scaling to hundreds of domains, dynamic expert pruning, or fully joint optimization of both adapters as future work [2505.07690]. MoDER reframes continual adaptation as modular expert recomposition for unseen classes and future domains [2508.16463]. Earlier tensor and low-rank approaches point toward adaptive rank control, compression of the shared core, and hybridization with routing or mask-based methods [1904.06345][2207.09074]. Collectively, these directions suggest that MTIL is evolving from a narrow anti-forgetting problem into a broader study of how fixed foundation models, selective routing, and compact per-task modules can jointly support retention, transfer, and zero-shot generalization across arbitrarily shifting domains.

Source: https://www.emergentmind.com/topics/multi-domain-task-incremental-learning-mtil