Multi-Expert Feature Learning (MEFL)

• MEFL is a design pattern where multiple expert networks learn specialized and complementary feature representations to enhance prediction accuracy.
• It underpins diverse applications such as object detection, domain generalization, lifelong learning, and cross-modal registration through adaptive routing.
• Architectural variations in MEFL, including dynamic gating, feature concatenation, and ensemble methods, offer trade-offs between computational efficiency and performance.

Searching arXiv for papers on Multi-Expert Feature Learning and closely related multi-expert methods. arXiv search: "multi-expert feature learning" Multi-Expert Feature Learning (MEFL) denotes a family of architectures in which multiple experts learn complementary feature spaces, local subspaces, or specialized representations, and a routing, gating, concatenation, averaging, or distillation mechanism combines them for prediction. Across the literature, MEFL appears in regularized mixture-of-experts models for local feature selection, dynamic task-specific decoding with memory experts, representation-level and time-axis aggregation for class-imbalanced learning, expert-guided domain generalization, transformed-view fusion for cross-modal registration, and expert-assigned detection pipelines (Peralta, 2014, Ye et al., 2023, Lee et al., 28 May 2026, Chen et al., 2024, Wang et al., 6 Jul 2025, Lee et al., 2017). This suggests that MEFL is best understood not as a single architecture but as a recurring design pattern for specialization, adaptive routing, and feature fusion.

1. Conceptual origins and formal foundations

An early precursor framed classification as a multi-agent system with a CenterAgent and class-specific expert agents, where each expert maintained K-region, M-region, and D-region features with associated probabilities and updated them through online consultation. In that system, feature learning was distributed across agents, and the CenterAgent used a “data-header like feature set” to decide which experts to consult for a query object (0902.2751). Although this formulation predates current deep multi-expert models, it already combined specialization, selective consultation, and online feature revision.

A more explicit statistical foundation appears in a regularized mixture-of-experts formulation. There, the conditional model is

$$p(y \mid x)=\sum_{i=1}^K p(y \mid x,m_i)\,p(m_i\mid x),$$

with multinomial-logit experts and a multinomial-logit gate. Responsibilities are defined as

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$

and sparsity in the expert parameters $\omega_{\ell ij}$ and gate parameters $\nu_{ij}$ induces local feature selection for experts and gates (Peralta, 2014). In that framework, $L_1$ regularization yields expert-specific and gate-specific feature sparsity, while an additional selector $\mu_{in}$ supports per-sample expert selection through

$$p(m_i \mid x_n)=\frac{\exp\big(\mu_{in}\nu_i^\top x_n\big)}{\sum_{j=1}^K \exp\big(\mu_{jn}\nu_j^\top x_n\big)}.$$

This formalization remains important because it isolates two mechanisms that recur throughout later MEFL systems: feature sparsity within each expert, and input-dependent control over which experts are active. A plausible implication is that later MEFL variants differ mainly in how they instantiate these two ideas—through gates, assignment networks, concatenation, replay, or distillation—rather than in their underlying objective of learning distinct but composable feature structures.

2. Expert assignment, gating, and routing mechanisms

In object detection, ME R-CNN operationalizes expert assignment at the RoI level. It is equipped with multiple experts where each expert is learned to process a certain type of regions of interest, and a learnable expert assignment network (EAN) automatically learns the optimal RoI-expert relationship even without any supervision of expert assignment. At test time, each RoI is directed to the expert with the largest EAN score, while training uses a staged procedure tailored to optimize the experts, EAN, and the shared network in an end-to-end fashion (Lee et al., 2017). Here routing is not merely a soft combination layer; it actively determines which expert-specific feature transformation is used for each proposal.

TaskExpert generalizes routing to multi-task visual scene understanding by decomposing backbone features into several representative task-generic features and then decoding task-specific features with dynamic task-specific gating networks. For layer $l>1$, the task-specific feature is

$$\mathbf{F}^{l,t}=\sum_{i=1}^K \mathbf{G}^{l,t}_i \odot \mathbf{R}^l_i+\mathbf{G}^{l,t}_{K+1}\odot \mathbf{M}^{l-1,t},$$

where $\mathbf{M}^{l-1,t}$ is a multi-task feature memory that acts as an additional expert (Ye et al., 2023). The gating is therefore per-task, per-sample, and per-location, and the memory expert extends routing across layers rather than restricting it to a single stage of the network.

RegistrationMamba applies routing to transformed views rather than native branch identities. It generates transformed variants $$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$0, extracts features for each variant, recalibrates them by inverse transformation, and fuses them through a learnable soft router:

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$1

The default design uses four experts and an image transformation pool including flipping, random rotation, and homography transformation (Wang et al., 6 Jul 2025). In this setting, expert diversity is induced by geometry, and routing decides how much each transformed-view feature map contributes to the final representation.

A common simplification is to equate MEFL with sparse mixture-of-experts routing. The literature is broader. Some systems use dense soft fusion over all experts, some route at the level of RoIs, some route at the level of tasks or spatial tokens, and some introduce memory as an additional expert. This suggests that routing is a design axis within MEFL, not its sole defining property.

3. Feature aggregation and ensemble-style MEFL

DAMEL defines MEFL along two axes. Along the representation axis, multiple experts share an early backbone, produce normalized expert features $$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$2, and these are concatenated into

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$3

A single auxiliary balanced classifier is trained on top of the concatenated representation, while each expert is trained with standard cross-entropy on the imbalanced data. Along the time axis, network weights are aggregated across training epochs by epoch-level EMA,

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$4

and only the EMA weights are used for inference (Lee et al., 28 May 2026). MEFL here is explicitly feature-level rather than prediction-level: expert diversity is preserved in the concatenated representation instead of being averaged away before classification.

FeatDistill adopts a heterogeneous four-backbone Vision Transformer ensemble composed of 2 CLIP ViT-L/14 and 2 SigLIP So400M experts. Each expert is trained in two stages: first with a standard binary classification objective, then with dense feature-level self-distillation for representation alignment. At inference time, the final prediction is obtained by averaging the probabilities from four independently trained experts (Tu et al., 23 Mar 2026). In this configuration, MEFL emerges from backbone heterogeneity, pre-training differences, and dense feature stabilization rather than from an explicit gate.

These two systems clarify an important distinction. DAMEL uses expert features as a joint representation for one downstream classifier, whereas FeatDistill keeps experts independent and combines them by soft voting. Both are MEFL in the sense that multiple experts define the representational substrate of the predictor, but they differ sharply in where fusion occurs: before classification in DAMEL, after classification in FeatDistill. This suggests that MEFL spans both feature-space composition and decision-level ensembling, provided that the experts are organized around complementary feature learning.

4. Multi-task, domain-general, and continual variants

In domain generalization, LFME uses one expert per source domain together with a single target model that is used alone at inference time. The target is trained on the union of all source domains with

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$5

where expert probabilities act as professional guidance by regularizing the logit of the target model (Chen et al., 2024). This makes MEFL a training-time regularization mechanism rather than a test-time ensemble: multiple experts shape the learned representation of a single deployable model.

TaskExpert places MEFL in multi-task learning by learning multiple representative task-generic feature spaces and dynamically assembling task-specific features from them. The multi-task feature memory updates across layers and serves as an additional expert, so the model combines cross-layer memory with layer-local expert outputs under task-specific gating (Ye et al., 2023). The resulting architecture is neither a standard shared encoder with static decoders nor a conventional sparse MoE; it is a dynamic multi-expert decoder with memory-mediated long-range task modeling.

TAME moves MEFL into lifelong deep learning. It maintains a pool of pretrained experts $$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$6, freezes them during the lifelong phase, and trains only a shared dense layer. Expert selection is task-aware and hard:

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$7

A replay buffer stores images, labels, and expert features, and AE-TAME adds scaled dot-product attention over stored features before prediction (Wang et al., 12 Dec 2025). This variant treats experts as stable feature extractors and delegates adaptation to replay, attention, and the shared prediction head.

Taken together, these systems show that MEFL is not restricted to a single inference paradigm. Some models retain all experts at test time, some discard them, and some freeze experts while learning only lightweight integration layers. A plausible implication is that the central object in MEFL is the expert-defined feature space, while the deployment strategy remains application dependent.

5. Cross-modal, multimodal, and interpretability-oriented extensions

DEFT-LLM defines MEFL as disentanglement across three parallel expert encoders: a structural expert, a temporal expert, and a motion-semantics expert. Their outputs $$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$8 are projected into three prefix tokens

$$R_{in}=p(m_i\mid x_n,y_n)=\frac{p(y_n\mid x_n,m_i)\,p(m_i\mid x_n)}{\sum_{j=1}^K p(y_n\mid x_n,m_j)\,p(m_j\mid x_n)},$$9

which are prepended to the textual input of LLaMA-3.1-8B. Training uses

$\omega_{\ell ij}$0

with Uni-MER providing motion-driven instruction data that aligns text with local facial motion (Zhang et al., 14 Nov 2025). Here MEFL is explicitly disentangled and multimodal: the experts remain separate, and the LLM performs the final cross-modal reasoning over expert tokens.

In remote sensing, RegistrationMamba applies MEFL to cross-modal registration under severe nonlinear radiometric variations and limited textures. Multiple transformed image variants are processed by specialized experts and fused by a soft router, while the backbone uses multi-directional cross-scanning and a multi-level feature aggregation module to combine global contextual relationships with fine-grained local information (Wang et al., 6 Jul 2025). This makes MEFL a representation-enrichment layer for matching rather than classification.

A different extension appears in sparse autoencoders for mechanistic interpretability. Beyond Redundancy introduces Multiple Expert Activation, in which several experts are activated per token and a global TopK is taken across their activations:

$\omega_{\ell ij}$1

with reconstruction

$\omega_{\ell ij}$2

Feature Scaling then reparameterizes the encoder as

$\omega_{\ell ij}$3

amplifying high-frequency components to promote diversity and reduce redundancy (Xu et al., 7 Nov 2025). In this setting, MEFL is not primarily about downstream accuracy; it is about learning diverse, specialized, and interpretable feature dictionaries.

These cross-modal and interpretability-focused systems indicate that MEFL is not tied to a particular output modality. It can support LLM prompting, remote-sensing alignment, or sparse feature decomposition, as long as multiple experts are used to separate or enrich feature structure before final integration.

6. Empirical behavior, recurrent benefits, and limitations

Empirically, MEFL is repeatedly associated with gains over single-expert or less structured baselines. ME R-CNN improves Faster R-CNN from $\omega_{\ell ij}$4 to $\omega_{\ell ij}$5 on PASCAL VOC 2007 and from $\omega_{\ell ij}$6 to $\omega_{\ell ij}$7 when trained on VOC07+12; on COCO test-dev it improves mAP@[0.5:0.95] from $\omega_{\ell ij}$8 to $\omega_{\ell ij}$9 (Lee et al., 2017). RegistrationMamba on OS-256 improves CMR(3) from $\nu_{ij}$0 without MEFL to $\nu_{ij}$1 with MEFL, while $\nu_{ij}$2 drops from $\nu_{ij}$3 to $\nu_{ij}$4 (Wang et al., 6 Jul 2025). DAMEL reports that representation-level aggregation outperforms prediction-level aggregation on ImageNet-LT/ResNet-10, with 2 experts $\nu_{ij}$5 vs. $\nu_{ij}$6, 3 experts $\nu_{ij}$7 vs. $\nu_{ij}$8, and 4 experts $\nu_{ij}$9 vs. $L_1$0, and the full dual-axis model reaches $L_1$1 accuracy with bias$L_1$2 and variance $L_1$3 on CIFAR-100-LT $L_1$4 (Lee et al., 28 May 2026).

LFME improves average accuracy across five DomainBed classification datasets from $L_1$5 to $L_1$6 with ResNet-18 and from $L_1$7 to $L_1$8 with ResNet-50; combined with SWAD, it reaches $L_1$9 versus $\mu_{in}$0 for ERM+SWAD (Chen et al., 2024). In continual learning, TAME improves Average Forgetting in all 5 sequences, for example from $\mu_{in}$1 to $\mu_{in}$2 in Sequence 2 under FID-based routing, and AE-TAME further improves A-AUROC in several sequences (Wang et al., 12 Dec 2025). In sparse autoencoders, Beyond Redundancy reports a $\mu_{in}$3 lower reconstruction error and a $\mu_{in}$4 reduction in feature redundancy compared to existing MoE-SAE methods (Xu et al., 7 Nov 2025).

At the same time, the literature records recurring limitations. The regularized mixture-of-experts formulation that most directly links MEFL with local feature selection states that “The experiments are still pending,” so its role is mainly methodological (Peralta, 2014). DAMEL notes that hyperparameter choice, including the number of experts $\mu_{in}$5 and the EMA coefficient $\mu_{in}$6, is empirically tuned, and that there is no formal theory yet for the optimal number of experts or EMA schedules (Lee et al., 28 May 2026). LFME requires domain labels for training data and multiple source domains (Chen et al., 2024). TAME fixes the number of experts in advance and freezes all expert CNNs after pretraining, which reduces forgetting but may hinder adaptation to very different future tasks (Wang et al., 12 Dec 2025). RegistrationMamba improves accuracy but increases FLOPs and inference time as experts are added (Wang et al., 6 Jul 2025).

A common misconception is that MEFL is synonymous with sparse test-time expert routing. The surveyed systems show a wider space: dense gating with a memory expert in TaskExpert, concatenated representations with one auxiliary classifier in DAMEL, training-only experts in LFME, frozen expert encoders projected into an LLM in DEFT-LLM, and transformed-view soft fusion in RegistrationMamba (Ye et al., 2023, Lee et al., 28 May 2026, Chen et al., 2024, Zhang et al., 14 Nov 2025, Wang et al., 6 Jul 2025). This suggests that the most stable definition of MEFL is architectural rather than algorithmic: multiple experts are used to structure feature learning itself, and the downstream integration rule is chosen to match the task, compute budget, and supervision regime.