MoDER: Modular Embedding Recomposition

• MoDER is a continual learning method for vision-language models that enhances zero-shot classification by recomposing modular textual experts.
• It learns low-rank, class-specialized adapters (LoRA-style PEFT modules) that are stored in a foundational hub and linearly recombined for unseen classes.
• Evaluations on Class-IL and MTIL settings demonstrate significant improvements in CI-Transfer and overall efficiency with fewer trainable parameters.

Searching arXiv for the primary paper and closely related continual-learning/VLM work. MoDular Embedding Recomposition (MoDER) is a continual learning method for vision-LLMs (VLMs) that targets a specific limitation of incremental fine-tuning: prior methods largely aim to preserve the zero-shot behavior of pre-trained models such as CLIP, whereas MoDER explicitly seeks to enhance zero-shot classification on not-yet-seen classes as new tasks arrive. The method does so by learning modular, class-specialized textual experts, storing them in a foundational hub, and recomposing them at inference time to synthesize refined text prototypes for unseen classes. It is evaluated in zero-shot incremental settings spanning Class-Incremental Learning (Class-IL) and Multi-domain Task Incremental Learning (MTIL), across a total of 14 datasets (Panariello et al., 22 Aug 2025).

1. Problem formulation and scope

MoDER is defined in the setting of continual learning for pre-trained VLMs, with CLIP as the base model. The central tension is that CLIP’s pre-training yields strong zero-shot classification, but downstream fine-tuning can overwrite the alignment between visual and textual representations. In the continual setting, this produces two coupled failure modes: catastrophic forgetting on previously seen downstream classes and degradation of pre-trained zero-shot generalization to novel categories and domains (Panariello et al., 22 Aug 2025).

In Class-IL, tasks arrive sequentially as $t = 1,\dots,T$. Task $t$ provides a dataset $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$, with labels drawn from a disjoint class subset $\mathcal{Y}_t$, so that $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$ for $i \neq j$. At current task $t_c$, the seen-class set is

$Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$

The standard continual objective is written as

$$L_{\text{CL}} = \sum_{t=1}^{T} \frac{1}{N_t} \sum_{(x, y) \in \mathcal{D}_t} \left[ \mathcal{L}(f(x;\theta), y) \right],$$

but only current-task data are available during training, so direct minimization is impossible. At evaluation time, task identity is unknown and prediction is performed over all seen classes. MoDER additionally uses Class Incremental Transfer (CI-Transfer) to measure accuracy on future unseen classes.

In MTIL, there is a sequence of 11 tasks corresponding to distinct datasets or domains: Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, and SUN397. Task identity is known at test time, the total number of classes is 1,201, and the reported metrics are Transfer, Average, and Last.

A common misconception is to treat MoDER as a generic continual-learning regularizer. More precisely, it is a zero-shot continual-learning method whose objective is not merely non-degradation of the pre-trained model, but the extraction and later recombination of modular knowledge from each observed class. This suggests a shift from conservative preservation to cumulative augmentation of the text-side prototype space.

2. Core mechanism: modular textual experts and recomposition

The core idea is to train many parameter-efficient textual experts, each specialized in representing a single seen class, and to store them in a foundational hub. For an unseen class, MoDER queries the hub, retrieves the most semantically related experts in the original CLIP text space, and linearly recomposes them into a synthesized adapter. This mechanism is called Mixture of Textual Experts (MoTE) (Panariello et al., 22 Aug 2025).

The base VLM is CLIP, with visual encoder $f_{\text{img}}(\cdot)$ and text encoder $t$0. During MoDER training, the visual encoder is frozen, and the base text encoder is also frozen; only parameter-efficient modules are learned. In the reported experiments, CLIP ViT-L/14 (OpenAI) is used for Class-IL and CI-Transfer, and CLIP ViT-B/16 (OpenAI) for MTIL.

The textual experts are LoRA-style PEFT modules attached to the text encoder. For each seen class $t$1, MoDER defines a low-rank adapter $t$2. With class-specific prompt $t$3, the resulting text embedding is

$t$4

At the layer level, for weight matrix $t$5, the fine-tuned displacement is

$t$6

with $t$7, $t$8, and $t$9. The experiments use rank $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$0. The paper also evaluates VeRA as an alternative PEFT with similar usage and reduced memory.

The foundational hub $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$1 is the set of all learned experts $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$2 for all seen classes across tasks. It grows over time. Retrieval for an unseen class $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$3 is performed in the frozen CLIP text space. Let

$\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$4

and define cosine similarity

$\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$5

MoDER selects the top-$\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$6 most semantically similar seen classes, denoted $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$7, converts similarities into softmax-normalized affinity weights $\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$8, and constructs a recomposed adapter

$\mathcal{D}_t = \{(x^{(n)}, y^{(n)})\}_{n=1}^{N_t}$9

The unseen-class embedding is then

$\mathcal{Y}_t$0

This recomposed adapter is a synthesized unseen-class-specific perturbation built from previously learned class specialists.

MoDER is therefore not a conventional token-routed sparse MoE. Its modularity is located in class-specialized low-rank text-encoder offsets, and recomposition occurs in adapter parameter space rather than through dynamic expert routing over input tokens.

3. Training pipeline and optimization objectives

For each current task $\mathcal{Y}_t$1, MoDER follows a multi-stage training procedure. First, it uses the frozen image encoder to extract visual embeddings for all current-task images. Second, it trains a lightweight class-conditioned diffusion model $\mathcal{Y}_t$2 on those extracted visual features. Third, it constructs a synthetic dataset $\mathcal{Y}_t$3 by sampling embeddings from all per-task generators seen so far. Fourth, it performs Textual Alignment (TA) to train experts $\mathcal{Y}_t$4 for classes in $\mathcal{Y}_t$5 on the synthetic dataset. Finally, the trained experts are stored in the hub (Panariello et al., 22 Aug 2025).

The replay component is notable because it operates in feature space, not pixel space. For each task $\mathcal{Y}_t$6, the generator is a class-conditioned diffusion model implemented as an 8-layer MLP with 256 hidden units per layer and SELU activations. At the start of TA for task $\mathcal{Y}_t$7, synthetic visual embeddings are sampled from every generator $\mathcal{Y}_t$8 for $\mathcal{Y}_t$9, yielding

$\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$0

In practice, the method uses $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$1 embeddings per class and batch size 512.

Given a synthetic datapoint $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$2, where $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$3 is a visual embedding and $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$4 the class id, each class expert produces

$\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$5

and the corresponding similarity score is

$\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$6

MoDER then uses a sigmoid loss over all classes,

$\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$7

where $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$8 is $\mathcal{Y}_i \cap \mathcal{Y}_j = \emptyset$9 if $i \neq j$0, otherwise $i \neq j$1. For the correct class, the objective pushes similarity upward; for the others, it pushes similarity downward.

The paper gives two reasons for using sigmoid rather than cross-entropy. First, cross-entropy would require a joint forward/backward pass over all experts, increasing GPU and memory cost. Second, sigmoid allows each class to be treated as an independent binary classifier, which enables more efficient and distributed training. The paper further reports that this objective improves transfer to unseen classes, which suggests that independently specialized experts are more modular and more composable at recomposition time.

Two robustness mechanisms are integral to training and inference. Template Augmentation (TAug) randomly samples from the approximately 80 CLIP ImageNet templates during training, rather than using a single fixed prompt template. $i \neq j$2-smoothing blends each expert with the original zero-shot text encoder: $i \neq j$3 For an unseen class, the same principle applies to the recomposed adapter,

$i \neq j$4

This keeps the adapted encoder closer to the original CLIP behavior and preserves out-of-distribution robustness.

Only the PEFT parameters $i \neq j$5 and generators $i \neq j$6 are trained; the CLIP encoders remain frozen throughout.

4. Inference, classification, and evaluation protocols

At inference time, MoDER distinguishes between seen and unseen classes. For a seen class $i \neq j$7, it uses that class’s own smoothed expert: $i \neq j$8 These prototypes can be precomputed and cached. For an unseen class $i \neq j$9, it applies MoTE: it computes frozen-CLIP text embeddings for the unseen prompt and all seen prompts, ranks seen classes by cosine similarity, selects the top-$t_c$0 neighbors, computes softmax weights

$t_c$1

forms the recomposed adapter

$t_c$2

and then encodes $t_c$3 with the smoothed composite weights (Panariello et al., 22 Aug 2025).

Classification itself follows CLIP’s similarity-based decision rule. For a test image $t_c$4,

$t_c$5

and for every candidate class $t_c$6, the method computes

$t_c$7

The difference from vanilla CLIP zero-shot is that MoDER uses smoothed adapted experts for seen classes and recomposed experts for unseen classes, rather than relying exclusively on the original unadapted text encoder.

The benchmark structure is summarized below.

Setting	Composition	Metrics
Class-IL	5 datasets: Split ImageNet-R, Split Cars-196, Split CUB-200, Split EuroSAT, Split ISIC	Final Avg. Accuracy, CI-Transfer
MTIL	11 tasks: Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, SUN397	Transfer, Average, Last

For Class-IL, task identity is unknown at test time and prediction is over all seen classes. For MTIL, task identity is known and classification is within the current task’s label set. The Class-IL datasets include both fine-grained and shifted regimes, including domain-shifted ImageNet-R, satellite imagery in EuroSAT, and medical imaging in ISIC. This breadth is significant because MoDER’s claim is not confined to a single type of incremental stream.

5. Empirical findings, efficiency, and ablations

On Class-IL, MoDER’s main reported gain is on CI-Transfer, the zero-shot metric over future unseen tasks. The paper states that MoDER (LoRA) significantly outperforms all baselines, including ZSCL, MoE-Adapters, and CGIL, across datasets, with average CI-Transfer $t_c$8 versus CGIL $t_c$9. On seen classes, measured by Final Avg. Accuracy, it also slightly exceeds state-of-the-art methods such as CGIL and STAR-Prompt, with average $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$0 versus CGIL $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$1 (Panariello et al., 22 Aug 2025).

On MTIL, the reported results are:

• Transfer: CLIP 69.4, MoDER 69.7 $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$2
• Average: CLIP 65.3, MoDER 76.9 $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$3, MoE-Adapters 76.7, ZSCL 75.4
• Last: CLIP 65.3, MoDER 85.8 $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$4, MoE-Adapters 85.0, ZSCL 83.6

These values support the paper’s central distinction between preservation and enhancement. In MTIL, MoDER does not merely retain CLIP’s Transfer score; it slightly improves it while also leading on Average and Last.

The ablation study identifies three components as especially consequential. First, sigmoid versus cross-entropy: using cross-entropy produces substantially lower CI-Transfer, with average 67.1 versus 75.3. Second, Template Augmentation: removing TAug slightly reduces CI-Transfer, from 75.3 to 74.3. Third, $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$5-smoothing: setting $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$6 causes a large drop on OOD and unseen datasets, with averages falling to 67.2. The paper explicitly describes $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$7-smoothing as crucial for preserving OOD zero-shot performance while using experts.

The method is also reported as comparatively efficient. In MTIL, MoDER uses 20.9M trainable parameters and $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$8GB GPU memory, whereas LwF / ZSCL use $Y = \bigcup_{t=1}^{t_c} \mathcal{Y}_t.$9M parameters and 26–32GB, and MoE-Adapters use $$L_{\text{CL}} = \sum_{t=1}^{T} \frac{1}{N_t} \sum_{(x, y) \in \mathcal{D}_t} \left[ \mathcal{L}(f(x;\theta), y) \right],$$0M parameters and 22GB. The paper therefore characterizes MoDER as significantly more parameter- and memory-efficient than those baselines.

A practical implication is that MoDER’s gains are not derived from dual-model preservation schemes or large replay buffers. The method uses no real replay buffer, relying instead on synthetic feature replay through per-task generators. This does not remove all overhead, but it constrains memory growth relative to approaches that retain multiple CLIP instances or large reference datasets.

6. Relation to prior work, conceptual interpretation, and limitations

MoDER is situated among VLM-based continual-learning methods such as L2P, DualPrompt, CODA-Prompt, AttriCLIP, CGIL, ZSCL, MoE-Adapters, STAR-Prompt, Inf-LoRA, and SLCA. The paper distinguishes itself from these approaches in three ways: class-level modularity, a foundational hub of experts that can be linearly recombined, and an explicit goal of active enhancement of zero-shot capability rather than conservative maintenance (Panariello et al., 22 Aug 2025).

Relative to CGIL, MoDER also uses class-wise text-side specialization and synthetic visual features, but the treatment of unseen classes differs. CGIL falls back to handcrafted templates for unseen classes, whereas MoDER composes unseen-class-specific adapters from the hub. Relative to ZSCL, MoDER does not depend on a teacher–student dual-CLIP setup with a reference dataset. Relative to MoE-Adapters, it does not require task identity at inference and does not frame unseen-class handling as expert routing over task-conditioned adapters.

The paper also places MoDER in a broader modularity discourse. One nearby line is token-routed sparse modularity, exemplified by "EMO: Pretraining Mixture of Experts for Emergent Modularity" (Wang et al., 7 May 2026), which induces semantically coherent expert subsets through document-level routing constraints. Another is post hoc composition on a shared substrate, exemplified by "Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate" (Bochkov, 8 Jul 2025), where specialist models can be merged by logit averaging because they share a frozen embedding interface. MoDER differs from both: it does not learn sparse token-routing pools, and it does not merge full models at the logit level. Its recomposition occurs specifically through linear combinations of low-rank text-encoder perturbations associated with individual seen classes.

Conceptually, the method is motivated by task vectors and linearity in parameter space. Each expert $$L_{\text{CL}} = \sum_{t=1}^{T} \frac{1}{N_t} \sum_{(x, y) \in \mathcal{D}_t} \left[ \mathcal{L}(f(x;\theta), y) \right],$$1 is a low-rank displacement that improves alignment to one class. For an unseen class, MoTE combines the displacements of semantically related seen classes. This suggests that the original CLIP text space serves as a retrieval geometry, while the adapter space serves as a mechanism for data-driven correction of zero-shot prototypes. A plausible implication is that MoDER treats each observed class not only as a supervised target, but also as an increment to a reusable basis for future zero-shot synthesis.

The stated and implied limitations are equally important. The hub can grow large because the most granular form stores one expert per class; the paper notes task-level experts as a possible memory-saving variant. Expert selection depends heavily on the quality of the frozen CLIP text encoder; severe domain shift or weak pre-training could degrade the semantic-neighbor retrieval used by MoTE. The per-task diffusion generator is lightweight compared with full-image replay, but it still adds training complexity. The paper also notes the absence of formal guarantees for when linear recomposition of LoRA modules will help, and it observes that gains over strong baselines are not uniformly large on every dataset.

Implementation guidance in the paper reflects these design choices: start from a pre-trained CLIP with frozen image and text encoders; train per-task feature generators; construct synthetic replay from all generators; train LoRA or VeRA textual experts via Textual Alignment with random template sampling; apply $$L_{\text{CL}} = \sum_{t=1}^{T} \frac{1}{N_t} \sum_{(x, y) \in \mathcal{D}_t} \left[ \mathcal{L}(f(x;\theta), y) \right],$$2-smoothing for inference; cache seen-class prototypes; and recompose unseen-class experts via top-$$L_{\text{CL}} = \sum_{t=1}^{T} \frac{1}{N_t} \sum_{(x, y) \in \mathcal{D}_t} \left[ \mathcal{L}(f(x;\theta), y) \right],$$3 similarity in frozen CLIP text space. The public code is integrated into the Mammoth continual learning framework.

In summary, MoDER defines a modular view of continual adaptation in VLMs: each class contributes a specialized text-side adapter to a growing hub, and zero-shot inference for future classes is improved by recombining those adapters rather than by preserving the base model in place. Within continual learning for CLIP-like systems, that reframes incremental experience as a source of composable prototype refinement rather than a threat to be merely controlled.